Hearing and monitoring system

ABSTRACT

Systems and methods for assisting a user include a housing custom fitted to a user anatomy; a microphone to capture sound coupled to a processor to deliver enhanced sound to the user anatomy; an amplifier with gain and amplitude controls; and a learning machine (such as a neural network) to identify an aural environment and adjusting amplifier controls to optimize hearing based on the identified aural environment.

BACKGROUND

The preferred embodiment relates to in-ear monitoring systems.

Hearing aid appliances with custom fit are known. The “In the Ear(ITE),” “In The Canal” or “Completely In the Canal” types of hearingaids have typically been manufactured using manual labor to a very largeextent. Tto produce ITE hearing devices, the shape of the auditory canalto be transferred to the housing shell of an ITE hearing device isdetermined in as precise a manner as possible. An ear impression of anauditory canal is usually taken in order to produce the shape for ahousing shell therefrom. Ear impressions of this type have been readinto PC systems by means of scanning for some time, in order then to befurther processed digitally. These scanners mostly use the triangulationmethod for measuring the 3D data of the objects. To this end, a lightsource (projector) is used, which projects a pattern. This pattern isrecorded again by a camera, which is disposed at an angle from theprojector. The spatial depth structure can be calculated.

Methods are also known, in which the auditory canal is directly scanned,without requiring an ear impression. These scanners, like those used torecord an ear impression, are relatively expensive and are almostexclusively based on triangulation measurement methods, whichnecessitate precise optical systems and also require a complex systemadjustment. Normally, due to the natural curvature of the ear canal, theeardrum is not visible from outside the ear. In order to overcome thenatural curvature of the ear canal, a skilled operator or physician hasto carefully pull the outer ear upward and to the back while carefullypushing the tip of the scanner into the ear canal as deeply as necessaryto display the eardrum. The ear canal has to be deformed in such a waythat the physician has a free view onto the eardrum along the opticalaxis of the camera. The procedure needs manual skills and significanttraining to carefully push the funnel into the ear canal while lookinginside and manipulating the curvature of the ear canal by pulling theear. For example, the trained operator is well aware to brace the handholding of the camera against the subject's head to avoid injury to theear canal and the eardrum by placing the index finger or little fingeragainst the head. Thus, the risk of penetration of the sensitive earcanal skin or even of the eardrum exists. Other than pain andhandicapped hearing, such an injury is also known to potentially inducecardiovascular complication through vagal overstimulation and,therefore, has to be avoided under all circumstances.

SUMMARY

In one aspect, systems and methods for assisting a user include ahousing custom fitted to a user anatomy; a microphone to capture soundcoupled to a processor to deliver enhanced sound to the user anatomy; anamplifier with gain and amplitude controls for each hearing frequency;and a learning machine (such as a neural network) to identify an auralenvironment (such as party, movie, office, or home environment) andadjusting amplifier controls to optimize hearing based on the identifiedaural environment. In one embodiment, the environment can be identifiedby the background noise or inferred through GPS location, for example.

In another aspect, a method for assisting a user includes customizing anin-ear device to a user anatomy; capturing sound using the in-eardevice; enhancing sound based on predetermined profiles and transmittingthe sound to an ear drum.

In yet another aspect, a method for assisting a user includescustomizing an in-ear device to a user anatomy; capturing sound usingthe in-ear device; capturing vital signs with the in-ear device; andlearning health signals from the sound and the vital signs from thein-ear device.

In a further aspect, a method includes customizing an in-ear device to auser anatomy; capturing vital signs with the in-ear device; and learninghealth signals from the vital signs from the in-ear device.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing vital signs to detect biomarkers with the in-eardevice; correlating genomic disease markers with the detected biomarkersto predict health with the in-ear device.

In another aspect, a method includes customizing an in-ear device to auser anatomy; identifying genomic disease markers; capturing vital signsto detect biomarkers with the in-ear device; correlating genomic diseasemarkers with the detected biomarkers to predict health with the in-eardevice.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing accelerometer data and vital signs; controllinga virtual reality device or augmented reality device with accelerationor vital sign data from the in-ear device.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing heart rate, EEG or ECG signal with the in-eardevice; and determining user intent with the in-ear device. Thedetermined user intent can be used to control an appliance, or can beused to indicate interest for advertisers.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing heart rate, EEG/ECG signal or temperature datato detect biomarkers with the in-ear device; and predict health with thein-ear device data.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing sounds from an advertisement, capturing vitalsigns associated with the advertisement; and customizing theadvertisement to attract the user.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing vital signs associated with a situation;detecting user emotion from the vital signs; and customizing an actionbased on user emotion. In one embodiment, such detected user emotion isprovided to a robot to be more responsive to the user.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing a command from a user, detecting user emotionbased on vital signs; and performing an action in response to thecommand and the detected user emotion.

In another aspect, a method includes customizing an in-ear device to auser anatomy; capturing a command from a user, authenticating the userbased on a voiceprint or user vital signs; and performing an action inresponse to the command.

In one aspect, a method for assisting a user includes customizing anin-ear device to a user anatomy; capturing sound using the in-eardevice; enhancing sound based on predetermined profiles and transmittingthe sound to an ear drum.

In one aspect, a method for assisting a user includes providing anin-ear device to a user anatomy; capturing sound using the in-eardevice; capturing vital signs with the in-ear device; and learninghealth signals from the sound and the vital signs from the in-eardevice.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing vital signs with the in-ear device; and learninghealth signals from the vital signs from the in-ear device.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing vital signs to detect biomarkers with the in-eardevice; correlating genomic disease markers with the detected biomarkersto predict health with the in-ear device.

In another aspect, a method includes providing an in-ear device to auser anatomy; identifying genomic disease markers; capturing vital signsto detect biomarkers with the in-ear device; correlating genomic diseasemarkers with the detected biomarkers to predict health with the in-eardevice.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing accelerometer data and vital signs; controllinga virtual reality device or augmented reality device with accelerationor vital sign data from the in-ear device.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing heart rate, EEG or ECG signal with the in-eardevice; and determining user intent with the in-ear device. Thedetermined user intent can be used to control an appliance, or can beused to indicate interest for advertisers.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing heart rate, EEG/ECG signal or temperature datato detect biomarkers with the in-ear device; and predict health with thein-ear device data.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing sounds from an advertisement, capturing vitalsigns associated with the advertisement; and customizing theadvertisement to attract the user.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing vital signs associated with a situation;detecting user emotion from the vital signs; and customizing an actionbased on user emotion. In one embodiment, such detected user emotion isprovided to a robot to be more responsive to the user.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing a command from a user, detecting user emotionbased on vital signs; and performing an action in response to thecommand and the detected user emotion.

In another aspect, a method includes providing an in-ear device to auser anatomy; capturing a command from a user, authenticating the userbased on a voiceprint or user vital signs; and performing an action inresponse to the command.

In another aspect, a method includes providing an in-ear device to auser anatomy; determine an audio response chart for a user based on aplurality of environments (restaurant, office, home, theater, party,concert, among others), determining a current environment, and updatingthe hearing aid parameters to optimize the amplifier response to thespecific environment. The environment can be auto detected based on GPSposition data or external data such as calendaring data or can be userselected using voice command, for example. In another embodiment, alearning machine automatically selects an optimal set of hearing aidparameters based on ambient sound and other confirmatory data.

Implementations of any of the above aspects may include one or more ofthe following:

-   -   detecting electrical potentials encephalography (EEG) or        electrocardiogram (ECG) in the ear;    -   using a camera in the ear to detect ear health;    -   detecting blood flow with an in-ear sensor;    -   detecting with an in-ear sensor blood parameters including        carboxyhemoglobin (HbCO), methemoglobin (HbMet) and total        hemoglobin (Hbt);    -   detecting pressure based on a curvature of an ear drum;    -   detecting body temperature in the ear;    -   detecting one or more of: alpha rhythm, auditory steady-state        response (ASSR), steady-state visual evoked potentials (SSVEP),        visually evoked potential (VEP), visually evoked response (VER)        and visually evoked cortical potential (VECP), cardiac activity,        speech and breathing;    -   detecting alpha rhythm, auditory steady-state response (ASSR),        steady-state visual evoked potentials (SSVEP), and visually        evoked potential (VEP);    -   correlating EEG, ECG, speech and breathing to determine health;    -   correlating cardiac activity, speech and breathing;    -   determining user health by detecting fluid in an ear structure,        change in ear color, curvature of the ear structure;    -   determining one or more bio-markers from the vital signs and        indicating user health;    -   performing a 3D scan inside an ear canal;    -   matching predetermined points on the 3D scan to key points on a        template and morphing the key points on the template to the        predetermined points;    -   3D printing a model from the 3D scan and fabricating the in-ear        device;    -   correlating genomic biomarkers for diseases to the vital signs        from the in-ear device and applying a learning machine to use        the vital signs from the in-ear device to predict disease        conditions;    -   determining a fall based on accelerometer data, vital signs and        sound;    -   determining loneliness or mental condition based on activity of        life data; or    -   providing a user dashboard showing health data over a period of        time and matching research data on the health signals.

Advantages of the preferred embodiments may include one or more of thefollowing. By using the disclosed data analysis method, a health systemcan add a new method of collecting and analyzing patient information.Patients can be assured that there is quantitative, objectiveinformation. For most medical conditions, biomarker pattern can beobtained and compared against known conditions. The system suppliesadjunctive information by which the health practitioner may make abetter decision regarding treatment alternatives. Appropriatemeasurement yields tremendous information to the doctors and users. Thisinformation improves diagnosis, aids in better treatment, and allows forearlier detection of problems.

Features from any of the above-mentioned embodiments may be used incombination with one another in accordance with the general principlesdescribed herein. These and other embodiments, features, and advantageswill be more fully understood upon reading the following detaileddescription in conjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Other systems, methods, features, and advantages of the present systemwill be or will become apparent to one of ordinary skill in the art uponexamination of the following figures and detailed description. It isintended that all such additional systems, methods, features, andadvantages be included within this description, be within the scope ofthe present invention, and be protected by the accompanying claims.Component parts shown in the drawings are not necessarily to scale, andmay be exaggerated to better illustrate the important features of thepresent invention. In the drawings, like reference numerals designatelike parts throughout the different views, wherein:

FIG. 1 shows an exemplary process to render sound and to monitor a user.

FIGS. 2A-2B show a custom monitoring device.

FIG. 3A shows an exemplary scanner to scan ear structures.

FIG. 3B shows a smart-phone scanner.

FIG. 3C shows exemplary point clouds from the camera(s).

FIG. 3D shows another custom hearing aid and monitoring device.

FIGS. 4A-4B show another custom hearing aid and monitoring device.

FIGS. 5A-5B show another custom hearing aid and monitoring device withextended camera, microphone and antenna.

FIG. 6 shows exemplary audiogram test flow and user interface designs.

FIG. 7A shows a neural network, FIG. 7B shows an exemplary analysis withnormal targets and outliers as warning labels, FIG. 7C show an exemplarylearning system, FIG. 7D shows a medical AI system with the learningmachines, and FIG. 7E show an exemplary AI based expert consultationsystem with human healthcare personnel.

FIG. 8A-8C shows exemplary coaching system for skiing, bicycling, andweightlifting/free style exercise, respectively, while FIG. 8D shows akinematic modeling for detecting exercise motion which in turn allowsprecision coaching suggestions, all using data from ear sensors.

Similar reference characters denote corresponding features consistentlythroughout the attached drawings.

DETAILED DESCRIPTION OF THE INVENTION

A system to create a custom medical device insertable into the ear isdescribed. The system uses an ear scanner to create a 3D model of theear structure, and a 3D printer that reproduces the shape (or thenegative of the shape) of the ear, wherein an ear insert can be formedfrom the shape of the ear. Suitable sensors and transducers are thenadded to the custom ear insert. The result custom ear insert iscomfortable to wear and blocks out extraneous sound or noise.

FIG. 1 shows an exemplary process to form and use a custom ear insert asa hearing aid with vital sign monitoring. The process is as follows:

-   -   Scan ear (1)    -   Generate 3D model of ear structure (2)    -   3D Print a Negative of the ear structure with provisioned space        for electronic module (3)    -   Form a flexible ear insert from the ear structure negative (4)    -   Insert or add amplifier to the provisioned space to the ear        insert (5)    -   During use, snugly push the ear insert toward the ear canal (6)    -   Render sound (7)    -   Capture medical data as biomarkers including ECG, EGG, physical        condition, heart rate, blood flow data, temperature (8)

FIGS. 2A-2B show an exemplary ear insert that is injection molded with apliant material. In one embodiment, the material is a medical gradethermoplastic elastomer. The device can render sound for the user, andcan also collect medical data such as oxygenation or other bloodconstituents or temperature, for example.

Turning now to FIG. 2A, an ear monitoring device 10 includes acustomized shell 12 for reception in the ear canal of a person. Theshell 12 has an opening 13 to receive the hearing aid 10 when mounted.In general, the shell 12 has the frontally facing part pointing outwardfrom the person's ear canal when the shell is positioned therein and thefrontally facing part has an opening 13 to receive an electronic moduletherein. The part of the shell 12 actually engaging the user's ear canalis denoted 11.

Ambient sound from the environment is picked up and amplified by thedevice of FIG. 2A-2B. Based on a hearing test, the amplifier frequencyresponse is enhanced based on the ear structure and the user's hearingdeficiencies. For example, certain sound frequencies are not perceivedwell by certain users, and the system would compensate for suchfrequency hearing deficiency. Moreover, at user request, the frequencybandwidth can be adjusted to specific situations, for example, toselectively focus on bird sounds or voice from a particular person orspecies. The enhanced or amplified sound moves through the ear canal andcauses the eardrum to move. The eardrum vibrates with sound. Soundvibrations move through the ossicles to the cochlea. Sound vibrationscause the fluid in the cochlea to move. Fluid movement causes the haircells to bend. Hair cells create neural signals which are picked up bythe auditory nerve. Hair cells at one end of the cochlea send low pitchsound information and hair cells at the other end send high pitch soundinformation. The auditory nerve sends signals to the brain where theyare interpreted as sounds.

The body of the device may be generated using suitable 3D printingtechniques including printing of a positive or negative of the ear usingstereo lithography (“SLA”) or stereo laser sintering (“SLS”) productionmethods and/or injection molding techniques. In one embodiment, anegative of the body is 3D printed and silicone or rubber is providedusing injection molding. While an electronic module can be snap fit intothe opening 13 and occupy the provisioned space, the module can besecured with an intermediary frame. This frame may be fixed in theopening 13 by any suitable means, such as adhesives, welding (laserwelding, heat welding), soldering, or as a press fit. The intermediaryframe may be permanently or removably fixed in the opening 13. Anydimensional slack between the intermediary frame and the opening 13 maybe filled by adhesive or by a sealing member of any suitable type (foam,rubber or other resilient material or the like). An electronic module(not shown) is adapted to be removably fixed in the intermediary framein order to facilitate replacement or repair thereof. The electronicsmodule can be flexible electronics, as detailed in commonly ownedapplication Ser. No. 16/237,697, the content of which is incorporated byreference. This insertion of the electronic module may be a click fixingor a combination of a clicking of a rod or axis into the intermediaryframe and a subsequent rotation and fixing of the electronic module inthe intermediary frame. This manner of fixing may be that normally usedin the prior art when fixing an electronic module to a shell.

The discussion already given above in the introduction to thespecification provide the expert with a large number of designs,depending on the ear hearing aid and monitoring device or itsconfiguration, to jointly manufacture two or more pertinent elements bytwo- or multi-component injection molding, in particular also byovermolding and then to assemble them jointly into an integral part.

FIG. 3A shows an exemplary ear scanner 30 with a camera mounted on a tipthat is inserted into the ear canal that terminates at an ear drum 32.The camera captures images that are converted into point cloud, as shownin FIG. 3B. In one embodiment, as the user or operator passes the cameraof the scanner in the ear, a point cloud is generated and a 3D model ofthe ear is constructed. The 3D model is used to construct the device ofFIG. 2, and further used in the sound modeling process to optimizeelectronic performance for hearing.

The scanner 20 has camera(s) that pick up a plurality of images whichare then used to reconstruct a 3D model of the ear structure. The tip ofthe scanner 20 is made sufficiently large that it cannot accidentally beinserted too far into the ear canal as to puncture the ear drum orotherwise harm the ear. The optics of the scanner 20 has a lamp and alens and the camera view angle provides at least one eccentricobservation point and/or at least one light source (preferably both)into an ear canal of a subject's outer ear and capturing imaged from theeccentric position. In particular, an optical axis does not have tocorrespond to the longitudinal axis of the head portion. Instead, anoptical axis of the electronic imaging unit may be arranged radiallyoffset.

In particular, in many cases, the ear canal of the outer ear is notstraight-lined, but exhibits at least one curvature, especially at atransition area or transition point between soft connective tissue andhard bone confining the ear canal. The “corner” is provided by thiscurvature and the requirement to deform the subject's ear is eliminatedor greatly reduced. Furthermore, the inventive method avoids the risk ofinjury to the ear canal, in particular the bony part of the ear canal,or to the eardrum by allowing the use of otoscopes with a tip of thehead portion that exhibit significantly larger dimensions as compared toan otoscope according to the art. Thus, the risk of introducing the headportion of the camera too deeply into the subject's ear canal isconsiderably reduced so that cell phone user or a layperson can to usethe scanner 20. This cuts down on cost. An eccentric position orobservation point allows for “looking around the corner”. In particular,the eardrum can be observed in its entirety, even in case the distal tipof the camera is introduced only as far as a transition area betweensoft connective tissue and hard bone confining the ear canal. The largerthe radial offset, the better the view onto the eardrum, even in case adistal end of the camera is positioned only in a transition area betweensoft connective tissue and hard bone confining the ear canal.Preferably, capturing the at least one image is carried out from aneccentric observation point which is positioned closer than 1.5 mm, morepreferable closer than 1.0 mm, further preferred closer than 0.8 mm oreven closer than 0.7 mm or 0.6 mm to an inner lateral surface of the earcanal, especially with respect to a diameter of the ear canal in therange between 4.8 mm and 5.2 mm. A guide facility allows the camera inthe ear canal to be pivoted, rotated and moved along the ear canal. Thecurved ear canal can thus be recorded in a desired fashion.

Once at least one image has been captured by the at least on electronicimaging unit, object recognition and unambiguous object identification(e.g. distinguishing objects, such as earwax, hair, and the eardrum) canbe performed by determining brightness and/or color information of thepixels of the at least one captured image. Each pixel of the imageobtained by the electronic imaging unit is characterized by a numericalvalue corresponding to the brightness of that pixel and—if theelectronic imaging device comprises a color imaging device—also by anumerical value corresponding to the color of that pixel. Differentobjects can be identified e.g. by their typical color. The earwax orobstacles can be digitally removed from the 3D model, and the user canbe notified to clean the ear. Alternatively, a vacuum embedded in thetip can be used to suction or remove wax, hair, and other debris fromthe ear structures to clean and capture the 3D model at the same time.

With these features, the electronic imaging unit is suitable to captureat least two images from different positions within the ear canal, e.g.by relocating one single electronic imaging unit when placed in thesubject's ear canal and/or by providing images from two or moreelectronic imaging units positioned at different sites in the ear canal.Alternatively or additionally, the method may be based on theimplementation of at least one illumination unit which is adapted toilluminate objects within the ear canal from different positions (e.g.from two or more positions). Preferably, a combination of bothapproaches is realized by the inventive method, which allows capturingimages from different positions under differing illumination conditions.Such a mode of action allows for reliable identification of distinctobjects (e.g. the eardrum, particles of earwax, hair, etc. in thesubject's ear), as will be described in more detail below. Thereby, therisk of image misinterpretation and failure in object recognition issignificantly reduced.

In one embodiment, a three-dimensional video may be created with aseries of time-separated measurements. In another aspect, the camera maybe moved (in a translation, a rotation, or some combination of these) inorder to capture a larger area of interest or the entire ear structure,or in order to obtain measurements of occluded surfaces of the earstructure, or for any other reason. In such a motion-based imagingprocess, the relative positions of the camera, the ear structure, and/orthe medium 106 may be physically tracked with motion sensors or thelike, or the relative motion may be inferred using a three-dimensionalregistration process to spatially relate successive three-dimensionaldata sets to one another. Regardless of the particular methodology, itwill be readily appreciated that individual spatial measurements, orgroups of spatial measurements, may be combined to form a largerthree-dimensional model, and all such techniques that would be apparentto one of ordinary skill in the art for creating a three-dimensionalreconstruction are intended to fall within the scope of this disclosure.

In another aspect, a desktop computer may provide a user interface forcontrol and operation of the system of FIG. 3A or 3B, as well as toolsfor displaying ear structure measurements, displaying or manipulatingreconstructed three-dimensional models, and so forth. The computer mayalso support calibration of the camera in order to correct for, e.g.,variations in the camera, and related optics, or variations inconcentration of additives to the medium that absorb, scatter,attenuate, fluoresce, or otherwise impart various optical properties.For example and without limitation, it will be understood that one cancharacterize the camera using a calibration fixture or the like, priorto employing the camera. Calibration may, for example, include the useof an ear structure having a known shape and a known position, or theuse of a container for the medium having a known shape. A variety ofsuitable calibration techniques will be readily appreciated based uponthe use of known shapes, dimensions, surface patterns, and so forth, anyof which may be adapted to use with the imaging systems describedherein.

FIG. 3B shows a smart phone scanner embodiment. In this embodiment,scanned ear structures can be detected on any ARKit. When first run, theapp displays a box that roughly estimates the size of whateverreal-world objects appear centered in the camera view. Before scanning,the system determines a region of the world contains the ear structures.The ear is scanned, and the camera is moved to scan from differentangles. The app highlights parts of the bounding box to indicatesufficient scanning to recognize the object from the correspondingdirection.

To enable object detection in an AR session, load the reference objectsas ARReferenceObject instances, provide those objects for thedetectionObjects property of an ARWorldTrackingConfiguration, and run anARSession with that configuration. When ARKit detects one of yourreference ear structures, the session automatically adds a correspondingARObjectAnchor to its list of anchors. To respond to an object beingrecognized, implement an appropriate ARSessionDelegate,ARSCNViewDelegate, or ARSKViewDelegate method that reports the newanchor being added to the session.

A reference object encodes a slice of the internal spatial-mapping datathat ARKit uses to track a device's position and orientation. To enablethe high-quality data collection required for object scanning, run asession with ARObjectScanningConfiguration:

let configuration=ARObjectScanningConfiguration( )

configuration.planeDetection=.horizontal

sceneView.session.run(configuration, options: .resetTracking)

During your object-scanning AR session, scan the object from variousangles to make sure you collect enough spatial data to recognize it. (Ifyou're building your own object-scanning tools, help users walk throughthe same steps this sample app provides.)

After scanning, callcreateReferenceObject(transform:center:extent:completionHandler:) toproduce an ARReferenceObject from a region of the user environmentmapped by the session:

// Extract the reference object based on the position & orientation ofthe bounding box. sceneView.session.createReferenceObject( transform:boundingBox.simdWorldTransform, center: float3( ), extent:boundingBox.extent, completionHandler: { object, error in if letreferenceObject = object { // Adjust the object's origin with theuser-provided transform. self.scannedReferenceObject =referenceObject.applyingTransform(origin.simdTransform)self.scannedReferenceObject!.name = self.scannedObject.scanName if letreferenceObjectToMerge = ViewController.instance?.referenceObjectToMerge{ ViewController.instance?.referenceObjectToMerge = nil // Show activityindicator during the merge. ViewController.instance?.showAlert(title:″″, message: ″Merging previous scan into this scan...″, buttonTitle:nil) // Try to merge the object which was just scanned with the existingone. self.scannedReferenceObject?.mergeInBackground(with:referenceObjectToMerge, completion: { (mergedObject, error) in if letmergedObject = mergedObject { mergedObject.name =self.scannedReferenceObject?.name self.scannedReferenceObject =mergedObject ViewController.instance?.showAlert(title: ″Mergesuccessful″, message: ″The previous scan has been merged into thisscan.″, buttonTitle: ″OK″) creationFinishe(self.scannedReferenceObject)} else { print(″Error: Failed to merge scans.\(error?.localizedDescription ?? ″″)″) let message = ″″″ Merging theprevious scan into this scan failed. Please make sure that there issufficient overlap between both scans and that the lighting environmenthasn't changed drastically. Which scan do you want to use for testing?″″″ let thisScan = UIAlertAction(title: ″Use This Scan″, style:.default) {_ in creationFinished(self.scannedReferenceObject) } letpreviousScan = UIAlertAction(title: ″Use Previous Scan″, style:.default) {_ in self.scannedReferenceObject = referenceObjectToMergecreationFinished(self.scannedReferenceObject) }ViewController.instance?.showAlert(title: ″Merge failed″, message:message, actions: [thisScan, previousScan]) } }) } else {creationFinished(self.scannedReferenceObject) } } else { print(″Error:Failed to create reference object. \(error!.localizedDescription)″)creationFinished(nil) } })

After getting an ARReferenceObject, the system can use it for detectionand dimension determination.

As shown in the embodiment of FIG. 3C, a mesh model is formed from thedimensional data. The reconstruction starts with a rough alignment of adummy mesh model to the point cloud data. In order to simplifyintegration of the ear model into a head model at a later stage, thedummy mesh model is prepared such that it includes part of the head aswell. The mesh part of the head is cropped such that it comprises arough ear plane, which can be matched with an ear plane of the pointcloud. An exemplary dummy mesh model is represented in FIG. 3C.

The rough alignment of the dummy mesh model is split into two stages.First the model is aligned to the data in 3D. Then orientation and scaleof the model ear are adapted to roughly match the data. The first stagepreferably starts with extracting a bounding box for the ear. This canbe done automatically using ear detection techniques. Alternatively, theear bounding box extraction is achieved by simple user interaction. Fromone of the images used for reconstructing the ear, which contains alateral view of the human head, the user selects a rectangle around theear. Advantageously, the user also marks the top point of the ear on thehelix. These simple interactions avoid having to apply involved eardetection techniques. From the cropping region a bounding box around theear is extracted from the point cloud. From this cropped point cloud twoplanes are estimated, one plane HP for the head points and one plane EPfor the points on the ear using a RANSAC plane fit process.

The ear plane is mainly used to compute the transformation necessary toalign the ear plane of the mesh model with that of the point cloud. Thefit enables a simple detection of whether the point cloud shows the leftear or the right ear based on the ear orientation (obtained, forexample, from the user input) and the relative orientation of the earplane and the head plane. In addition, the fit further allows extractingthose points of the point cloud that are close to the ear plane. Fromthese points the outer helix line can be extracted, which simplifiesestimating the proper scale and the ear-center of the model. To thisend, from the extracted points of the point cloud a depth map of the earpoints is obtained. This depth map generally is quite good, but it maynonetheless contain a number of pixels without depth information. Inorder to reduce this number, the depth map is preferably filtered. Forexample, for each pixel without depth information the median value fromthe surrounding pixels may be computed, provided there are sufficientsurrounding pixels with depth information. This median value will thenbe used as the depth value for the respective pixel. A useful propertyof this median filter is that it does not smooth the edges from thedepth map, which is the information of interest. Edges are extractedfrom the filtered depth map. This may be done using a canny edgedetector. From the detected edges connected lines are extracted. Inorder to finally extract the outer helix, the longest connected line onthe right/left side for a left/right ear is taken as a starting line.This line is then down-sampled and only the longest part is taken. Thelongest part is determined by following the line as long as the anglebetween two consecutive edges, which are defined by three consecutivepoints, does not exceed a threshold. As a starting point, a smalldown-sampling factor is chosen and is then iteratively increased. Onlythe factor that gives the longest outer helix is kept. This techniqueallows “smoothing” the line, which could be corrupted by some outliers.It is further assumed that the helix is smooth and does not containabrupt changes of the orientation of successive edges, which is enforcedby the angle threshold. Depending on the quality of the data, the helixline can be broken. As a result, the first selected line may not spanthe entire helix bound.

With the foregoing information the rough alignment can be computed. Tothis end the model ear plane is aligned to the ear plane in the pointcloud. Then the orientation of the model ear is aligned with that of thepoint cloud ear by a rotation in the ear plane. Following the roughalignment a finer elastic transformation is applied in order to fit themesh model to the data points. This is a specific instance of anon-rigid registration technique. Since the ear is roughly planar andhence can be characterized well by its 2D structure, the elastictransformation is performed in two steps. First the ear is alignedaccording to 2D information, such as the helix line detected before.Then a guided 3D transformation is applied, which respects the 2Dconditions. The two steps will be explained in more detail in thefollowing. For model preparation an ear region of the mesh model isselected, e.g. by a user input. This selection allows classifying allmesh model vertices as belonging to the ear or to the head. An exampleof a selected ear region of a mesh model is shown in FIG. 13, where theear region is indicated by the non-transparent mesh.

For the non-rigid alignment the mesh model can be deformed to match thedata points by minimizing a morphing energy consisting of: apoint-to-point energy for a model vertex and its closest data-point; apoint-to-plane energy for a model vertex, its closest data-point, andthe normal of it; a global rigid transformation term; and a local rigidtransformation term. This allows an elastic transformation. However,this energy is adapted for the present solution, as will be describedbelow. Note that only the 2D locations of all the points in the earplane are considered.

Next, sensors are detailed. An ear site has the advantage of morequickly and more accurately reflecting oxygenation changes in the body'score as compared to peripheral site measurements, such as a fingertip.Conventional ear sensors utilize a sensor clip on the ear lobe. However,significant variations in lobe size, shape and thickness and the generalfloppiness of the ear lobe render this site less suitable for centraloxygen saturation measurements than the concha and the ear canal.Disclosed herein are various embodiments for obtaining noninvasive bloodparameter measurements from concha 120 and ear canal 130 tissue sites.

In another embodiment, a hologram scanner can be used. The device forrecording the spatial structure of at least one part of an ear canal orear impression uses a holography unit with a light source and by meansof which a hologram of the ear canal can be adjusted. A semitransparentdisk in the ear is used for separating the light beam from the lightsource into an illumination beam and a reference beam. A sensor recordsan object beam, which is produced by reflection of the illumination beamonto the part of the ear canal, together with the reference beam. Thehologram recording system can essentially be constructed within smallerdimensions than a conventional 3D scanner, which is based on thetriangulation principle. The hologram recording system can beconstructed within significantly smaller dimensions than the 3D scanner.The hologram sensor (CCD chip) does not require a front lens, since itdoes not record a mapping of an image, but instead interference patternson its surface

One aspect of an ear sensor optically measures physiological parametersrelated to blood constituents by transmitting multiple wavelengths oflight into a concha site and receiving the light after attenuation bypulsatile blood flow within the concha site. The ear sensor comprises asensor body, a sensor connector and a sensor cable interconnecting thesensor body and the sensor connector. The sensor body comprises a base,legs and an optical assembly. The legs extend from the base to detectorand emitter housings. An optical assembly has an emitter and a detector.The emitter is disposed in the emitter housing and the detector isdisposed in the detector housing. The legs have an unflexed positionwith the emitter housing proximate the detector housing and a flexedposition with the emitter housing distal the detector housing. The legsare moved to the flexed position so as to position the detector housingand emitter housing over opposite sides of a concha site. The legs arereleased to the unflexed position so that the concha site is graspedbetween the detector housing and emitter housing.

Pulse oximetry systems for measuring constituents of circulating bloodcan be used in many monitoring scenarios. A pulse oximetry system has anoptical sensor applied to a patient, a monitor for processing sensorsignals and displaying results and a patient cable electricallyinterconnecting the sensor and the monitor. A pulse oximetry sensor haslight emitting diodes (LEDs), typically one emitting a red wavelengthand one emitting an infrared (IR) wavelength, and a photodiode detector.The emitters and detector are in the ear insert, and the patient cabletransmits drive signals to these emitters from the monitor. The emittersrespond to the drive signals to transmit light into the fleshy tissue.The detector generates a signal responsive to the emitted light afterattenuation by pulsatile blood flow within the fingertip. The patientcable transmits the detector signal to the monitor, which processes thesignal to provide a numerical readout of physiological parameters suchas oxygen saturation (SpO2) and pulse rate. Advanced physiologicalmonitoring systems may incorporate pulse oximetry in addition toadvanced features for the calculation and display of other bloodparameters, such as carboxyhemoglobin (HbCO), methemoglobin (HbMet) andtotal hemoglobin (Hbt), as a few examples. In other embodiments, thedevice has physiological monitors and corresponding multiple wavelengthoptical sensors capable of measuring parameters in addition to SpO2,such as HbCO, HbMet and Hbt are described in at least U.S. patentapplication Ser. No. 12/056,179, filed Mar. 26, 2008, titled MultipleWavelength Optical Sensor and U.S. patent application Ser. No.11/366,208, filed Mar. 1, 2006, titled Noninvasive Multi-ParameterPatient Monitor, both incorporated by reference herein. Further,noninvasive blood parameter monitors and corresponding multiplewavelength optical sensors to sense SpO2, pulse rate, perfusion index(PI), signal quality (SiQ), pulse variability index (PVI), HbCO andHbMet among other parameters.

Heart pulse can be detected by measuring the dilation and constrictionof tiny blood vessels in the ear canal. In one embodiment, the dilationmeasurement is done optically and in another embodiment, amicromechanical MEMS sensor is used. ECG sensor can be used where theelectrode can detect a full and clinically valid electrocardiogram,which records the electrical activity of the heart.

One example embodiment uses the Samsung Bio-Processor which integratesmultiple AFEs (Analog Front Ends) to measure diverse biometrics,including bioelectrical impedance analysis (BIA), photoplethysmogram(PPG), electrocardiogram (ECG), skin temperature and galvanic skinresponse (GSR) in a single chip solution. With integratedmicrocontroller unit (MCU), digital signal processor (DSP) and real-timeclock (RTC), the Bio-Processor can monitor data in the ear with lowpower requirement.

Impact sensors, or accelerometers, measure in real time the force andeven the number of impacts that players sustain. Data collected is sentwirelessly via Bluetooth to a dedicated monitor on the sidelines, whilethe impact prompts a visual light or audio alert to signal players,coaches, officials, and the training or medical staff of the team. Onesuch sensor example is the ADXL377 from Analog Devices, a small, thinand low-power 3-axis accelerometer that measures acceleration frommotion, shock, or vibration. It features a full-scale range of ±200 g,which would encompass the full range of impact acceleration in sports,which typically does not exceed 150 g's. When a post-impact individualis removed from a game and not allowed to return until cleared by aconcussion-savvy healthcare professional, most will recover quickly. Ifthe injury is undetected, however, and an athlete continues playing,concussion recovery often takes much longer. Thus, the system avoidsproblems from delayed or unidentified injury can include: Earlydementia, Depression, Rapid brain aging, and Death. The cumulativeeffects of repetitive head impacts (RHI) increases the risk of long-termneuro-degenerative diseases, such as Parkinson's disease, Alzheimer's,Mild Cognitive Impairment, and ALS or Lou Gehrig's disease. The sensors'most important role is to alert to dangerous concussions. Yet, the actof real-time monitoring brings these players to the attention of theircoaches not only to monitor serious impacts but, based on the dataprovided by the sensors, also help to modify a player's technique sothat they are not, for example, keeping their head low where they cansustain injury to the front and top of the skull. In the NFL there alsohas been an aggressive crackdown against hits to the head and neck—aresponse to the ongoing concussion crisis—resulting in immediate penaltyto players using their helmets as a “weapon”. Customized mouthguardsalso have sensors therein. A customized mouthguard has tested to be 99percent accurate in predicting serious brain injury afternear-concussive force, according to an Academy of General Dentistrystudy. Teeth absorb and scatter infrared light, which shows how muchforce is taking place at the moment of impact.

The device can use optical sensors for heart rate (HR) as a biomarker inheart failure (HF) both of diagnostic and prognostic values. HR is adeterminant of myocardial oxygen demand, coronary blood flow, andmyocardial performance and is central to the adaptation of cardiacoutput to metabolic needs. Increased HR can predict adverse outcome inthe general population and in patients with chronic HF. Part of theability of HR to predict risk is related to the forces driving it,namely, neurohormonal activation. HR relates to emotional arousal andreflects both sympathetic and parasympathetic nervous system activity.When measured at rest, HR relates to autonomic activity during arelaxing condition. HR reactivity is expressed as a change from restingor baseline that results after exposure to stimuli. Thesestress-regulating mechanisms prepare the body for fight or flightresponses, and as such can explain individual differences topsychopathology. Thus, the device monitors HR as a biomarker of bothdiagnostic and prognostic values.

The HR output can be used to analyze heart-rate variability (HRV) (thetime differences between one beat and the next) and HRV can be used toindicate the potential health benefits of food items. Reduced HRV isassociated with the development of numerous conditions for example,diabetes, cardiovascular disease, inflammation, obesity and psychiatricdisorders. Aspects of diet that are viewed as undesirable, for examplehigh intakes of saturated or trans-fat and high glycaemic carbohydrates,have been found to reduce HRV. The consistent relationship between HRV,health and morbidity allows the system to use HRV as a biomarker whenconsidering the influence of diet on mental and physical health. FurtherHRV can be used as a biomarker for aging. HRV can also act as biomarkersfor:

-   -   Overtraining: “Cumulative or too intensive sporting activity        (e.g. competition series, overtraining syndrome), however,        brings about a decrease in HRV”    -   Physical Fitness: “People who have an active lifestyle and        maintain a good or high level of physical fitness or        above-average sporting activity can achieve an increase in their        basic parasympathetic activity and thus an increase in their        HRV.”    -   Overweight: “an elevated body weight or elevated free-fat mass57        correlates with a decrease in HRV. Both active and passive        smoking lead to an increase in HRV”    -   Alcohol Abuse: “Regular chronic alcohol abuse above the alcohol        quantity of a standard drink for women or two standard drinks        for men reduces HRV, while moderate alcohol consumption up to        these quantities does not change the HRV and is not associated        with an increase”    -   Smoking: “Both active and passive smoking lead to an increase in        HRV”    -   Sleep: Another important factor that affects your HRV score is        the amount and quality of sleep.

In one embodiment, the system determines a dynamical marker ofsino-atrial instability, termed heart rate fragmentation (HRF) and isused a dynamical biomarker of adverse cardiovascular events (CVEs). Inhealthy adults at rest and during sleep, the highest frequency at whichthe sino-atrial node (SAN) rate fluctuates varies between ˜0.15 and 0.40Hz. These oscillations, referred to as respiratory sinus arrhythmia, aredue to vagally-mediated coupling between the SAN and breathing. However,not all fluctuations in heart rate (HR) at or above the respiratoryfrequency are attributable to vagal tone modulation. Under pathologicconditions, an increased density of reversals in HR acceleration sign,not consistent with short-term parasympathetic control, can be observed.

The system captures ECG data as biomarkers for cardiac diseases such asmyocardial infarction, cardiomyopathy, atrioventricular bundle branchblock, and rhythm disorders. The ECG data is cleaned up, and the systemextracts features by taking quantiles of the distributions of measureson ECGs, while commonly used characterizing feature is the mean. Thesystem applies commonly used measurement variables on ECGs withoutpreselection and use dimension reduction methods to identify biomarkers,which is useful when the number of input variables is large and no priorinformation is available on which ones are more important. Threefrequently used classifiers are used on all features and todimension-reduced features by PCA. The three methods are from classicalto modern: stepwise discriminant analysis (SDA), SVM, and LASSO logisticregression.

In one embodiment, four types of features are considered as inputvariables for classification: T wave type, time span measurements,amplitude measurements, and the slopes of waveforms for features such as

-   -   (1) T Wave Type. The ECGPUWAVE function labels 6 types of T        waves for each beat: Normal, Inverted, Positive Monophasic,        Negative Monophasic, Biphasic Negative-Positive, and Biphasic        Positive-Negative based on the T wave morphology. This is the        only categorical variable considered.    -   (2) Time Span Measurements. Six commonly used time span        measurements are considered: the length of the RR interval, PR        interval, QT interval, P wave, QRS wave, and T wave.    -   (3) Amplitude Measurements. The amplitudes of P wave, R-peak,        and T wave are used as input variables. To measure the P wave        amplitude, we first estimate the baseline by taking the mean of        the values in the PR segment, ST segment, and TP segment (from        the end of the T wave to the start of the P wave of the next        heartbeat), then subtract the maximum and minimum values of the        P wave by the estimated baseline, and take the one with a bigger        absolute value as the amplitude of P wave. Other amplitude        measurements are obtained similarly.    -   (4) The Slopes of Waveforms. The slopes of waveforms are also        considered to measure the dynamic features of a heartbeat. Each        heartbeat is split into nine segments and the slope of the        waveform in each segment is estimated by simple linear        regression.

The device can include EEG sensors which measure a variety of EEGresponses—alpha rhythm, ASSR, SSVEP and VEP—as well as multiplemechanical signals associated with cardiac activity, speech andbreathing. EEG sensors can be used where electrodes provide low contactimpedance with the skin over a prolonged period of time. A low impedancestretchable fabric is used as electrodes. The system captures variousEEG paradigms: ASSR, steady-state visual evoked potential (SSVEP),transient response to visual stimulus (VEP), and alpha rhythm. The EEGsensors can predict and assess the fatigue based on the neural activityin the alpha band which is usually associated with the state of wakefulrelaxation and manifests itself in the EEG oscillations in the 8-12 Hzfrequency range, centered around 10 Hz. The loss of alpha rhythm is alsoone of the key features used by clinicians to define the onset of sleep.A mechanical transducer (electret condenser microphone) within itsmultimodal electro-mechanical sensor, which can be used as a referencefor single-channel digital denoising of physiological signals such asjaw clenching and for removing real-world motion artifacts from ear-EEG.In one embodiment, a microphone at the tip of the earpiece facingtowards the eardrum can directly capture acoustic energy traveling fromthe vocal chords via auditory tube to the ear canal. The output of sucha microphone would be expected to provide better speech quality than thesealed microphone within the multimodal sensor.

The system can detect auditory steady-state response (ASSR) as abiomarker a type of ERP which can test the integrity of auditorypathways and the capacity of these pathways to generate synchronousactivity at specific frequencies. ASSRs are elicited by temporallymodulated auditory stimulation, such as a train of clicks with a fixedinter-click interval, or an amplitude modulated (AM) tone. After theonset of the stimulus, the EEG or MEG rapidly entrains to the frequencyand phase of the stimulus. The ASSR is generated by activity within theauditory pathway. The ASSR for modulation frequencies up to 50 Hz isgenerated from the auditory cortex based on EEG. Higher frequencies ofmodulation (>80 Hz) are thought to originate from brainstem areas. Thetype of stimulus may also affect the region of activation within theauditory cortex. Amplitude modulated (AM) tones and click train stimuliare commonly used stimuli to evoke the ASSR.

The EEG sensor can be used as a brain-computer interface (BCI) andprovides a direct communication pathway between the brain and theexternal world by translating signals from brain activities into machinecodes or commands to control different types of external devices, suchas a computer cursor, cellphone, home equipment or a wheelchair. SSVEPcan be used in BCI due to high information transfer rate (ITR), littletraining and high reliability. The use of in-ear EEG acquisition makesBCI convenient, and highly efficient artifact removal techniques can beused to derive clean EEG signals.

The system can measure visually evoked potential (VEP), visually evokedresponse (VER) or visually evoked cortical potential (VECP). They referto electrical potentials, initiated by brief visual stimuli, which arerecorded from the scalp overlying visual cortex, VEP waveforms areextracted from the electro-encephalogram (EEG) by signal averaging. VEPsare used primarily to measure the functional integrity of the visualpathways from retina via the optic nerves to the visual cortex of thebrain. VEPs better quantify functional integrity of the optic pathwaysthan scanning techniques such as magnetic resonance imaging (MRI). Anyabnormality that affects the visual pathways or visual cortex in thebrain can affect the VEP. Examples are cortical blindness due tomeningitis or anoxia, optic neuritis as a consequence of demyelination,optic atrophy, stroke, and compression of the optic pathways by tumors,amblyopia, and neurofibromatosis. In general, myelin plaques common inmultiple sclerosis slow the speed of VEP wave peaks. Compression of theoptic pathways such as from hydrocephalus or a tumor also reducesamplitude of wave peaks.

A bioimpedance (BI) sensor can be used to determine a biomarker of totalbody fluid content. The BIA is a noninvasive method for evaluation ofbody composition, easy to perform, and fast, reproducible, andeconomical and indicates nutritional status of patients by estimatingthe amount of lean body mass, fat mass, body water, and cell mass. Themethod also allows the assessment of patient's prognosis through the PA,which has been applied in patients with various diseases, includingchronic liver disease. The phase angle varies according to thepopulation and can be used for prognosis.

In another embodiment, the BI sensor can estimate glucose level. This isdone by measuring the bioimpedance at various frequencies, where highfrequency Bi is related to fluid volume of the body and low frequency BIis used to estimate the volume of extracellular fluid in the tissues.

The step of determining the amount of glucose can include comparing themeasured impedance with a predetermined relationship between impedanceand blood glucose level. In a particular embodiment, the step ofdetermining the blood glucose level of a subject includes ascertainingthe sum of a fraction of the magnitude of the measured impedance and afraction of the phase of the measured impedance. The amount of bloodglucose, in one embodiment, is determined according to the equation:Predicted glucose=(0.31)Magnitude+(0.24)Phase where the impedance ismeasured at 20 kHz. In certain embodiments, impedance is measured at aplurality of frequencies, and the method includes determining the ratioof one or more pairs of measurements and determining the amount ofglucose in the body fluid includes comparing the determined ratio(s)with corresponding predetermined ratio(s), i.e., that have beenpreviously correlated with directly measured glucose levels. Inembodiments, the process includes measuring impedance at two frequenciesand determining the amount of glucose further includes determining apredetermined index, the index including a ratio of first and secondnumbers obtained from first and second of the impedance measurements.The first and second numbers can include a component of said first andsecond impedance measurements, respectively. The first number can be thereal part of the complex electrical impedance at the first frequency andthe second number can be the magnitude of the complex electricalimpedance at the second frequency. The first number can be the imaginarypart of the complex electrical impedance at the first frequency and thesecond number can be the magnitude of the complex electrical impedanceat the second frequency. The first number can be the magnitude of thecomplex electrical impedance at the first frequency and the secondnumber can be the magnitude of the complex electrical impedance at thesecond frequency. In another embodiment, determining the amount ofglucose further includes determining a predetermined index in which theindex includes a difference between first and second numbers obtainedfrom first and second of said impedance measurements. The first numbercan be the phase angle of the complex electrical impedance at the firstfrequency and said second number can be the phase angle of the complexelectrical impedance at the second frequency.

The electrodes can be in operative connection with the processorprogrammed to determine the amount of glucose in the body fluid basedupon the measured impedance. In certain embodiments, the processorwireless communicates with an insulin pump programmed to adjust theamount of insulin flow via the pump to the subject in response to thedetermined amount of glucose. The BIA electrodes can be spaced betweenabout 0.2 mm and about 2 cm from each other.

In another aspect, the BI sensor provides non-invasive monitoring ofglucose in a body fluid of a subject. The apparatus includes means formeasuring impedance of skin tissue in response to a voltage appliedthereto and a microprocessor operatively connected to the means formeasuring impedance, for determining the amount of glucose in the bodyfluid based upon the impedance measurement(s). The means for measuringimpedance of skin tissue can include a pair of spaced apart electrodesfor electrically conductive contact with a skin surface. Themicroprocessor can be programmed to compare the measured impedance witha predetermined correlation between impedance and blood glucose level.The apparatus can include means for measuring impedance at a pluralityof frequencies of the applied voltage and the program can include meansfor determining the ratio of one or more pairs of the impedancemeasurements and means for comparing the determined ratio(s) withcorresponding predetermined ratio(s) to determine the amount of glucosein the body fluid.

In a particular embodiment, the apparatus includes means for calibratingthe apparatus against a directly measured glucose level of a saidsubject. The apparatus can thus include means for inputting the value ofthe directly measured glucose level in conjunction with impedancemeasured about the same time, for use by the program to determine theblood glucose level of that subject at a later time based solely onsubsequent impedance measurements.

One embodiment measures BI at 31 different frequencies logarithmicallydistributed in the range of 1 kHz to 1 Mhz (10 frequencies per decade).Another embodiment measures BI a t two of the frequencies: 20 and 500kHz; and in the second set of experiments, 20 kHz only. It may be foundin the future that there is a more optimal frequency or frequencies. Itis quite possible, in a commercially acceptable instrument thatimpedance will be determined at at least two frequencies, rather thanonly one. For practical reasons of instrumentation, the upper frequencyat which impedance is measured is likely to be about 500 kHz, but higherfrequencies, even has high as 5 MHz or higher are possible and areconsidered to be within the scope of this invention. Relationships maybe established using data obtained at one, two or more frequencies.

One embodiment, specifically for determining glucose levels of asubject, includes a 2-pole BI measurement configuration that measuresimpedance at multiple frequencies, preferably two well spaced apartfrequencies. The instrument includes a computer which also calculatesthe index or indices that correlate with blood glucose levels anddetermines the glucose levels based on the correlation(s). an artificialneural network to perform a non-linear regression.

In another embodiment, a BI sensor can estimate sugar content in humanblood based on variation of dielectric permeability of a finger placedin the electrical field of transducer. The amount of sugar in humanblood can also be estimate by changing the reactance of oscillatingcircuits included in the secondary circuits of high-frequency generatorvia direct action of human upon oscillating circuits elements. With thismethod, the amount of sugar in blood is determined based on variation ofcurrent in the secondary circuits of high-frequency generator. Inanother embodiment, a spectral analysis of high-frequency radiationreflected by human body or passing through the human body is conducted.The phase shift between direct and reflected (or transmitted) waves,which characterizes the reactive component of electrical impedance,represents a parameter to be measured by this method. The concentrationof substances contained in the blood (in particular, glucoseconcentration) is determined based on measured parameters of phasespectrum. In another embodiment, glucose concentration is determined bythis device based on measurement of human body region impedance at twofrequencies, determining capacitive component of impedance andconverting the obtained value of capacitive component into glucoseconcentration in patient's blood. Another embodiment measures impedancebetween two electrodes at a number of frequencies and deriving the valueof glucose concentration on the basis of measured values. In anotherembodiment, the concentration of glucose in blood is determined basedmathematical model.

The microphone can also detect respiration. Breathing creates turbulencewithin the airways, so that the turbulent airflow can be measured usinga microphone placed externally on the upper chest at the suprasternalnotch. The respiratory signals recorded inside the ear canal are weak,and are affected by motion artifacts arising from a significant movementof the earpiece inside the ear canal. A control loop involving knowledgeof the degree of artifacts and total output power from the microphonescan be used for denoising purposes from jaw movements. Denoising can bedone for EEG, ECG, PPG waveforms.

An infrared sensor unit can detect temperature detection in conjunctionwith an optical identification of objects allows for more reliableidentification of the objects, e.g. of the eardrum. Providing the deviceadditionally with an infrared sensor unit, especially arrangedcentrically at the distal tip, allows for minimizing any risk ofmisdiagnosis.

In one implementation information relating to characteristics of thepatient's tympanic cavity can be evaluated or processed. In this casethe electronics includes a camera that detects serous or mucous fluidwithin the tympanic cavity can be an indicator of the eardrum itself,and can be an indicator of a pathologic condition in the middle ear.Within the ear canal, only behind the eardrum, such body fluid can beidentified. Thus, evidence of any body fluid can provide evidence of theeardrum itself, as well as evidence of a pathologic condition, e.g. OME.

In a method according to the preferred embodiment, preferably, anintensity of illumination provided by the at least one light source isadjusted such that light emitted by the at least one light source isarranged for at least partially transilluminating the eardrum in such away that it can be reflected at least partially by any object or bodyfluid within the subject's tympanic cavity arranged behind the eardrum.The preferred embodiment is based on the finding that translucentcharacteristics of the eardrum can be evaluated in order to distinguishbetween different objects within the ear canal, especially in order toidentify the eardrum more reliably. Thereby, illumination can beadjusted such that tissue or hard bone confining the ear canal isoverexposed, providing reflections (reflected radiation or light),especially reflections within a known spectrum, which can be ignored,i.e. automatically subtracted out. Such a method enables identificationof the eardrum more reliably.

In particular, the degree of reddishness or reflectivity of light in thered spectral range can be determined at different illuminationintensities. It can therefore be distinguished more reliably betweenlight reflected by the eardrum itself, or by objects or fluids behindthe eardrum, or by the mucosal covering the tympanic cavity wall. Thereflectivity of light may be evaluated with respect to reflectivitywithin e.g. the green or blue spectral range. Typical spectralwavelength maxima are 450 nm (blue light), 550 nm (green light), and 600nm (red light) for a respective (color) channel. The electronic imagingunit, e.g. comprising a color video camera, or any color sensitivesensor, may record images with respect to the red, green or bluespectral range, respectively. A logic unit may calculate, compare andnormalize brightness values for each read, green and blue image,especially with respect to each separate pixel of the respective image.Such an evaluation may also facilitate medical characterization of theeardrum. In particular, the healthy eardrum is a thin, semitransparentmembrane containing only few relatively small blood vessels. Incontrast, an inflamed eardrum may exhibit thickening and/or increasedvascularization. Also, any skin or tissue confining the ear canal aswell as any mucosa in the middle ear may be heavily vascularized. Inother words: The reflectivity in the different spectral ranges variesconsiderably between the different structures or objects as well asbetween healthy and inflamed tissue. Thus, referring to the spectralrange enables more reliable differentiation between light reflected bythe eardrum itself, or by objects or any fluid behind the eardrum, or bythe tympanic cavity wall covered by mucosa.

Thereby, the risk of confounding any red (inflamed) section of the earcanal and the eardrum can be minimized. Also, the eardrum can beidentified indirectly by identifying the tympanic cavity. In particular,any opaque fluid, especially amber fluid containing leukocytes andproteins, within the tympanic cavity may influence the spectrum ofreflected light, depending on the intensity of illumination. At arelatively high intensity of illumination, the spectrum of reflectedlight will be typical for scattering in serous or mucous fluidcontaining particles like leukocytes, as light transmits the eardrum andis at least partially reflected by the opaque fluid. At a relatively lowintensity of illumination, the spectrum of reflected light will bedominated by the eardrum itself, as a considerable fraction of the lightdoes not transmit the eardrum, but is directly reflected by the eardrum.Thus, information relating to the tympanic cavity, especially moredetailed color information, can facilitate identification of the eardrumas well as of pathologic conditions in the middle ear.

Transilluminating the eardrum can provide supplemental information withrespect to the characteristics of the eardrum (e.g. the shape,especially a convexity of the eardrum), and/or with respect to thepresence of any fluid within the tympanic cavity. Spectral patterns ofreflected light which are typical for eardrum reflection and tympaniccavity reflection can be use to determine the area of interest as wellas a physiologic or pathologic condition of the eardrum and the tympaniccavity, especially in conjunction with feedback controlled illumination.

Any fluid within the tympanic cavity evokes a higher degree ofreflection than the physiologically present air. The fluid increasesreflectance. In contrast, in case the tympanic cavity is filled withair, any light transilluminating the eardrum is only reflected withinferior intensity, as most of the light is absorbed within the tympaniccavity. In other words: transilluminating the eardrum and evaluatingreflected light in dependence on the intensity of illumination canfacilitate determining specific characteristics of the eardrum, e.g. anabsolute degree of reflectivity in dependence on different wavelengthsand intensities, providing more information or more certain informationwith respect to the type of tissue and its condition. Evaluatingreflected light can comprise spectral analysis of translucentreflection, especially at different illumination intensities.

The degree of reflection in the red spectrum from the area of theeardrum may depend on the illumination level, i.e. the intensity ofillumination. In particular, the red channel reflection can increasewith increasing intensity of illumination. The higher the intensity ofillumination, the higher the red channel reflection intensity. Also, ithas been found that at relatively high intensities of illumination, notonly the eardrum, but also any other tissue will reflect more light inthe red spectrum. Therefore, on the one hand, providing a control orlogic unit which is arranged for adjusting the intensity of illuminationcan facilitate identification of the eardrum. On the other hand, it canfacilitate determining specific characteristics of the eardrum, e.g. anabsolute degree of red channel reflection, such that the red channelreflection provides more information or more certain information withrespect to the type of tissue and state of the tissue.

The degree of red channel reflection does not increase in the samemanner with increasing intensity of illumination, depending on thepresence of body fluid behind the eardrum. It has been found that incase there is body fluid within the tympanic cavity, with increasingintensity of illumination, the degree of red channel reflection does notincrease as strongly as if the tympanic cavity was empty. Thus, based onthe (absolute) degree of red channel reflection, the presence of fluidbehind the eardrum can be evaluated. This may facilitate determinationof pathologic conditions, e.g. OME.

The camera and process can identify pattern recognition of geometricalpatterns, especially circular or ellipsoid shapes, or geometricalpatterns characterizing the malleus bone, or further anatomicalcharacteristics of the outer ear or the middle ear. Pattern recognitionallows for more reliable identification of the eardrum. Patternrecognition can comprise recognition based on features and shapes suchas the shape of e.g. the malleus, the malleus handle, the eardrum orspecific portions of the eardrum such as the pasr flaccida or thefibrocartilagenous ring. In particular, pattern recognition may compriseedge detection and/or spectral analysis, especially shape detection of acircular or ellipsoid shape with an angular interruption at the malleusbone or pars flaccida.

In a method according to the preferred embodiment, preferably, themethod further comprises calibrating a spectral sensitivity of theelectronic imaging unit and/or calibrating color and/or brightness ofthe at least one light source. Calibration allows for more reliableidentification of objects. It has been found that in case the lightintensity is very high allowing for passing light through a healthyeardrum, which is semitransparent, a considerable amount of light withinthe red spectrum can be reflected by the tympanic cavity (especially dueto illumination of red mucosa confining the middle ear). Thus,calibrating brightness or the intensity of emitted light enables moreaccurate evaluation of the (absolute) degree of red channel reflectionand its source. In other words, spectral calibration of the imagingsensor in combination with spectral calibration of the illuminationmeans allows for the evaluation of the tissue types and conditions.

Calibration can be carried out e.g. based on feedback illuminationcontrol with respect to different objects or different kinds of tissue,once the respective object or tissue has been identified. Thereby,spectral norm curves with respect to different light intensities providefurther data based on which calibration can be carried out.

FIG. 3D shows an earpiece 50 that has one or more sensors 52, aprocessor 54, a microphone 56, and a speaker 58. The earpiece 50 may beshaped and sized for an ear canal of a subject. The transducer 52 may beany of the previously discussed sensors (EEG, ECG, camera, temperature,pressure, among others). In general, the sensor 52 may be positionedwithin the earpiece at a position that, when the earpiece 50 is placedfor use in the ear canal, corresponds to a location on a surface of theear canal that exhibits a substantial shape change correlated to amusculoskeletal movement of the subject. The position depicted in FIG.3B is provided by way of example only, and it will be understood thatany position exhibiting substantial displacement may be used to positionthe sensor(s) 52 for use as contemplated herein. In one aspect, thesensor 52 may be positioned at a position that, when the earpiece isplaced for use in the ear canal, corresponds to a location on a surfaceof the ear canal that exhibits a maximum surface displacement from aneutral position in response to the musculoskeletal movement of thesubject. In another aspect, the transducer 52 may be positioned at aposition that, when the earpiece is placed for use in the ear canal,corresponds to a location on a surface of the ear canal that exceeds anaverage surface displacement from a neutral position in response to themusculoskeletal movement of the subject. It will be understood that,while a single transducer 52 is depicted, a number of transducers may beincluded, which may detect different musculoskeletal movements, or maybe coordinated to more accurately detect a single musculoskeletalmovement.

The processor 54 may be coupled to the microphone 56, speaker 58, andsensor(s) 52, and may be configured to detect the musculoskeletalmovement of the subject based upon a pressure change signal from thetransducer 52, and to generate a predetermined control signal inresponse to the musculoskeletal movement. The predetermined controlsignal may, for example, be a mute signal for the earpiece, a volumechange signal for the earpiece, or, where the earpiece is an earbud foran audio player (in which case the microphone 56 may optionally beomitted), a track change signal for the audio player coupled to theearpiece.

Power for the unit can be from a battery or scavenged from theenvironment using solar or temperature differential power generation. Inone embodiment, a biological battery can be tapped. Located in the partof the ear called the cochlea, the battery chamber is divided by amembrane, some of whose cells are specialized to pump ions. An imbalanceof potassium and sodium ions on opposite sides of the membrane, togetherwith the particular arrangement of the pumps, creates an electricalvoltage. A storage device receives charge that gradually builds upcharge in a capacitor. The voltage of the biological battery fluctuates,for example one circuit needs between 40 seconds and four minutes toamass enough charge to power a radio. The frequency of the signal wasthus itself an indication of the electrochemical properties of the innerear.

FIG. 4A shows a deployed device 30 that is plugged into a sound cable.The device has an outer face 32 that includes a microphone or microphonearray and energy scavenger on one side. The other side is an output port34 that is snugged fitted to the ear canal using a custom ear insertsuch as that of FIG. 2A-2B.

FIG. 4B shows in more detail the device 30 having an outer face 32 thatcan include microphone or microphone array. Further, the device canharvest energy with solar cells on the face 32. The unit can optionallyconnect to a remote microphone, camera, cellular transceiver, and/orbattery via cable 34. The cable or output port 34 can be connected to abehind-the-ear clip-on housing, for example. The device 30 has a customform protrusion or extension 36 that is custom to the wearer's earstructures. The extension 36 can include hearing aid electronics and/orbody sensors as detailed above. The sensors can help detect ear issuessuch as:

Chronic disease. Some cases of hearing loss are not caused by a problemwith the ear, but by an interruption of blood flow to the ear or brain.Strokes, heart disease, high blood pressure, diabetes, and rheumatoidarthritis can all cause mild to moderate hearing loss.

Meniere's disease. If the user is experiencing extreme dizziness, lossof balance, and nausea, a hearing screening could lead to a diagnosis ofMeniere's disease. This condition is caused by an imbalance of fluids inthe inner ear, causing a ringing in the ears (tinnitus), a blockedfeeling or hearing loss in one or both ears, and severe vertigo.

Paget's disease. This bone disorder may have no early symptoms, andcause lifelong injuries and medical conditions in the patient. As timegoes on, patients with Paget's disease may suffer hearing loss andchronic headaches, as well as nerve, bone, and joint pain. In severecases, patients may have abnormally large head sizes, improper spinecurvature, or severe bowing of the arms and legs.

Pendred syndrome. Pendred syndrome is a genetic condition that causeshearing loss, thyroid dysfunction, and balance problems in children. Achild who is born with Pendred syndrome is likely to lose hearingfunction early in life, in some cases before the child reaches threeyears old. Hearing loss caused by Pendred syndrome will usually worsenover time, and can lead to total deafness.

Otosclerosis. This disease causes the bones in the middle ear to harden,preventing them from conducting sound into the inner ear. Otosclerosiscan often be treated or even reversed with surgery.

In one particular variation for treating tinnitus, device may utilize anaudio signal, such as music and in particular music having a dynamicsignal with intensities varying over time with multiple peaks andtroughs throughout the signal. Other audio signals such as varioussounds of nature, e.g., rainfall, wind, waves, etc., or other signalssuch as voice or speech may alternatively be used so long as the audiosignal is dynamic. This audio signal may be modified according to amasking algorithm and applied through the device 14 and to the patientto partially mask the patient's tinnitus. An example of how an audiosignal may be modified is described in detail in U.S. Pat. No. 6,682,472(Davis), which is incorporated herein by reference in its entirety anddescribes a tinnitus treatment which utilizes software to spectrallymodify the audio signal in accordance with a predetermined maskingalgorithm which modifies the intensity of the audio signal at selectedfrequencies. The described predetermined masking algorithm providesintermittent masking of the tinnitus where the tinnitus is completelymasked during peaks in the audio signal and where the perceived tinnitusis detectable to the patient during troughs in the audio signal. Such analgorithm provides for training and habituation by the patient of theirtinnitus. Accordingly, the intensity of the audio signal may be modifiedacross the spectrum of the signal and may also be modified to accountfor any hearing loss that the patient may have incurred. The audiosignal having a dynamic spectrum with varying intensities. The audiosignal may completely mask the patient's tinnitus during peaks in thesignal while during troughs in the audio signal, the tinnitus may beperceived by the patient. Moreover, the masking algorithm may bemodified to account for any hearing loss of the patient.

FIGS. 5A-5B show an AR version where the earpiece 30 includes a 5Gtransceiver in the unit 30 that is connected to antenna, mike and camerain an extension 39, which in turn projects from the outer face 32. Theextension 39 is semi-rigid and can be adjusted by the user to aim atpredetermined areas. In one embodiment, a plurality of extensions 39extend like whiskers or hairs that provides signal reception but hard tosee. In other embodiments, a single extension 39 includes antenna(s)wrapped on the outside of the extension while images can be captured andtransmitted over fiber optics to a high resolution imager. Soundvibration can also be optically captured where incident sound wavesmodulate the light guided in optical fibers without the help ofelectricity. Intensity modulation by beam deflection at a movingmembrane. For example, an optical mike from PKI can be used.

To supplement the whisker antenna, the front and edge of the earpiece 30has 3D printed MIMO antennas for Wifi, Bluetooth, and 5G signals. Theextension 39 further includes a microphone and camera at the tip tocapture audio visual information to aid the user as an augmented realitysystem. The earpiece contains an inertial measurement unit (IMU) coupledto the intelligent earpiece. The IMU is configured to detect inertialmeasurement data that corresponds to a positioning, velocity, oracceleration of the intelligent earpiece. The earpiece also contains aglobal positioning system (GPS) unit coupled to the earpiece that isconfigured to detect location data corresponding to a location of theintelligent earpiece. At least one camera is coupled to the intelligentearpiece and is configured to detect image data corresponding to asurrounding environment of the intelligent guidance device.

In one embodiment, the earpiece 30 is a standalone 5G phone controlledby a machine computer interface through the EEG sensor as detailedabove. In another variation, voice command can be issued to the 5Gphone. In another variation, a foldable display is provided with alarger battery and 5G transceiver. When the user opens the foldabledisplay, the data processing including deep learning operations, 5Gtransceiver communication, IMU and GPS operations are automaticallytransferred to the foldable display to conserve power.

The earpiece for providing social and environmental awareness cancontinuously observe the user and his surroundings as well as storepreference information, such as calendars and schedules, and accessremote databases. Based on this observed data, the earpiece canproactively provide feedback to the user. Proactive functions can, forexample, remind a user where he should be, inform the user of the nameof a person he is speaking with, warn the user when the user may beapproaching a hazardous situation, etc. This is advantageous over thestate of the art because the user of the earpiece can be providedinformation without having to request it. This can result in the userbeing provided feedback that he may not have known he could receive.Additionally, it allows the user to receive feedback without wastingextra time or effort. In some circumstances, this proactive feedback canprevent potential embarrassment for the user (for example, he need notask the earpiece the name of a person he is speaking with). When leftand right earpieces are deployed, the stereo microphones and cameras ofthe system provide depth and distance information to the device. Thedevice can then use this information to better determine social andenvironmental elements around the user. The combination of the globalpositioning system (GPS), the inertial measurement unit (IMU) and thecamera is advantageous as the combination can provide more accuratefeedback to the user. In one embodiment, the earpiece relies on the GPSand IMU and 5G streams from a smart phone to minimize size and powerconsumption. The earpiece can use smart phone memory to store objectdata regarding previously determined objects. The memory also storespreviously determined user data associated with the user. The earpiecealso includes a processor connected to the IMU, the GPS unit and the atleast one camera. The processor is configured to recognize an object inthe surrounding environment. This is done by analyzing the image databased on the stored object data and at least one of the inertialmeasurement data or the location data. The processor is also configuredto determine a desirable event or action based on the recognized object,the previously determined user data, and a current time or day. Theprocessor is also configured to determine a destination based on thedetermined desirable event or action. The processor is also configuredto determine a navigation path for navigating the intelligent guidancedevice to the destination. This is determined based on the determineddestination, the image data, and at least one of the inertialmeasurement data or the location data. The processor is also configuredto determine output data based on the determined navigation path. Aspeaker is included that is configured to provide audio information tothe user based on at least one of the recognized object, determineddesirable event or action, or navigation path. The AR system providescontinuous social and environmental awareness by the earpiece. Themethod includes detecting, via an inertial measurement unit (IMU), aglobal position system unit (GPS) or a camera, inertial measurement datacorresponding to a positioning, velocity, or acceleration of theearpiece. Location data corresponding to a location of the earpiece orimage data corresponding to a surrounding environment of the earpiece isalso determined. The method also includes storing, in a memory, objectdata regarding previously determined objects and previously determineduser data regarding the user. The method also includes recognizing, by aprocessor, an object in the surrounding environment by analyzing theimage data based on the stored object data and at least one of theinertial measurement data or the location data. The method furtherincludes determining, by the processor, a desirable event or actionbased on the recognized object, the previously determined user data, anda current time or day. The processor also determines a destination basedon the determined desirable event or action. The processor may determinea navigation path for navigating the intelligent guidance device to thedestination based on the determined destination, the image data, and atleast one of the inertial measurement data or the location data. Theprocessor may determine output data based on the determined navigationpath. The method further includes providing, via a speaker or avibration unit, audio or haptic information to the user based on atleast one of the recognized object, the determined desirable event oraction, or the navigation path. In one embodiment, the earpiece of FIGS.5A-5B also includes an antenna configured to transmit the image data,the inertial measurement data, the location data and the object data toa remote processor and to receive processed data from the remoteprocessor (such as a 5G smart phone or a remote server array, amongothers). The remote processor is configured to recognize an object inthe surrounding environment by analyzing the image data based on thestored object data and at least one of the inertial measurement data orthe location data. The remote processor is also configured to determinea desirable event or action based on the recognized object, thepreviously determined user data, and a current time or day. The remoteprocessor is also configured to determine a destination based on thedetermined desirable event or action. The remote processor is alsoconfigured to determine a navigation path for navigating the intelligentguidance device to the destination based on the determined destination,the image data, and at least one of the inertial measurement data or thelocation data. The remote processor is further configured to determineoutput data based on the determined navigation path. The earpiece alsoincludes a speaker configured to provide audio information to the userbased on at least one of the recognized object, determined desirableevent or action, or navigation path. For example, the user may give avoice command, “Take me to building X in Y campus.” The earpiece mayinstruct the smart phone download a relevant map if not already stored,or may navigate based on perceived images from the stereo cameras. Asthe user follows the navigation commands from the earpiece 30, the usermay walk by a coffee shop in the morning, and the earpiece 100 wouldrecognize the coffee shop and the time of day, along with the user'shabits, and appropriately alert the user. The earpiece 30 may verballyalert the user through the speaker. The user may use an input device ora web page to adjust settings, which for example may control the typesof alerts, what details to announce, and other parameters which mayrelate to object recognition or alert settings. The user may turn on oroff certain features as needed. When navigating indoors, the GPS may notprovide enough information to a blind user to navigate around obstaclesand reach desired locations or features. The earpiece cameras mayrecognize, for instance, stairs, exits, and restrooms and appropriatelymatch the images with the GPS mapping system to provide better guidance.

The earpiece 30 can respond to a request which may be, for example, arequest to identify a person, identify a room, identify an object,identify any other place, navigate to a certain location such as anaddress or a particular room in a building, to remind the user of hiscurrent action, what color an object is, if an outfit matches, whereanother person is pointing or looking. For other example, when thedetected data suggests that the user requires an opinion of anotherperson, a communication channel may be established with a device ofanother person. For example, when the detected speech regarding anoutfit of the user, facial recognition data regarding the user beingindecisive or wondering about what to wear, and/or perceived action of auser in front of a mirror indicate that the user needs fashion advicefrom another person, a video teleconference between the user and afriend of the user may be established. From priorconversations/interactions, the earpiece may have previously stored auser's friend's contact information. The processor may categorize typesof friends of the user and recognize that this communication needs to bewith a friend that the user is comfortable with. The processor mayoutput data to the user letting the user know that a video conference orteleconference will be established with the friend. The earpiece 30 mayprovide a video connection to a friend of the user or send a picture ofthe outfit to a friend of the user. In this example, the friend mayprovide a response as to whether or not the outfit matches. The friendmay also assist the user in finding an alternate outfit that matches.

In order to program a hearing aid to be tailored to the user's hearingneeds, the user's hearing threshold may be measured using asound-stimulus-producing device and calibrated headphone. Themeasurement of the hearing threshold may take place in a sound-isolatingroom. For example, the measurement may occur in a room where there isvery little audible noise. The sound-stimulus-producing device and thecalibrated headphones may be referred to as an audiometer.

As shown in FIG. 6, the audiometer may generate pure tones at variousfrequencies between 125 Hz and 12,000 Hz that are representative of thefrequency bands in which the tones are included. These tones may betransmitted through the headphones of the audiometer to the individualbeing tested. The intensity or volume of the pure tones is varied untilthe individual can just barely detect the presence of the tone. For eachpure tone, the intensity of the tone at which the individual can justbarely detect the presence of the tone is known as the individual's airconduction threshold of hearing. The collection of the thresholds ofhearing at each of the various pure tone frequencies is known as anaudiogram and may be presented in graphical form.

After audio equipment calibration, each frequency will be testedseparately, at increasing levels. In one embodiment, the system startswith the lowest amplitude (quietest file at −5 dbHL for example) andstop when a user hearing threshold level has been reached. Fileslabelled 70 dBHL and above, are meant to detect severe hearing losses,and will play very loud for a normal hearing person. The equipmentcaptures responses used to generate an audiogram which is a graph thatshows the softest sounds a person can hear at different frequencies. Itplots the threshold of hearing relative to an average ‘normal’ hearing.In this test, the ISO 389-7:2005 standard is used. Levels are expressedin deciBels Hearing Level (dBHL). The system saves the measurements andcorresponding plots of Audiogram, Distortion, Time Analysis,Spectrogram, Audibility Spectrogram, 2-cc Curve, Occlusion Effects, andFeedback Analysis. A hearing aid prescription based on a selectedfitting prescription formula/rational can be filled. The selectedhearing aid can be adjusted and results analyzed and plotted with orwithout the involvement of the hearing-impaired individual. The systemcan optimize, objectively and subjectively, the performance of aselected hearing aid according to measured in-the-ear-canal proberesponse as a function of the selected signal model, hearing aidparameter set, the individual's measured hearing profile, and subjectiveresponses to the presented audible signal. The system can determine thecharacteristics of a simulated monaural or binaural hearing aid systemthat produces natural sound perception and improved sound localizationability to the hearing impaired individual. This is accomplished byselecting a simulated hearing aid transfer function that produces, inconjunction with the face-plate transfer function, a combined transferfunction that matches that of the unaided transfer function for eachear. The matching requirement typically involves frequency and phaseresponses. However, the magnitude response is expected to vary becausemost hearing impaired individuals require amplification to compensatefor their hearing losses.

Based on the audiogram, amplifier parameters can be adjusted to improvehearing. In one embodiment for obtaining hearing enhancement fittingsfor a hearing aid device is described. In one embodiment, a plurality ofaudiograms is divided into one or more sets of audiograms. Arepresentative audiogram is created for each set of audiograms. Ahearing enhancement fitting is computed from each representativeaudiogram. A hearing aid device is programmed with one or more hearingenhancement fittings computed from each representative audiogram. In oneembodiment, the one or more sets of audiograms may be subdivided intoone or more subsets until a termination condition is satisfied. In oneconfiguration, one or more audiograms may be filtered from the pluralityof audiograms. For example, one or more audiograms may be filtered fromthe plurality of audiograms that exceed a specified fitting range forthe hearing aid device. In one embodiment, a mean hearing threshold maybe determined at each measured frequency of each audiogram within theplurality of audiograms. Prototype audiograms may be created from themean hearing threshold. In addition, each prototype audiogram may beassociated with a set of audiograms. In one configuration, an audiogrammay be placed in the set of audiograms if the audiogram is similar tothe prototype audiogram associated with the set. In one embodiment, thecreation of a representative audiogram for each set of audiograms mayinclude calculating a mean of each audiogram in a set of audiograms.When the threshold of hearing in each frequency band has beendetermined, this threshold may be used to estimate the amount ofamplification, compression, and/or other adjustment that will beemployed in the hearing aid device to compensate for the individual'sloss of hearing. In the example of FIG. 6, the system will start at 500Hz with progressing loudness before going to the next row at 1 kHz, 2,3, 4, 5 and 20 kHz, respectively.

In one aspect, a method includes providing an in-ear device to a useranatomy; determine an audio response chart for a user based on aplurality of environments (restaurant, office, home, theater, party,concert, among others), determining a current environment, and updatingthe hearing aid parameters to optimize the amplifier response to thespecific environment. The environment can be auto detected based on GPSposition data or external data such as calendaring data or can be userselected using voice command, for example. In another embodiment, alearning machine automatically selects an optimal set of hearing aidparameters based on ambient sound and other confirmatory data.

In the example of FIG. 6, the system will start at 500 Hz withprogressing loudness before going to the next row at 1 kHz, 2, 3, 4, 5and 20 kHz, respectively, and then repeats the test in different ambientenvironments such as restaurant, office, home, theater, party, concert,among others and records the best audio responses in view of the“noise”. This is how the learning machine or neural network learns theoptimum settings based on the specific setting. Moreover, as the useroperates the device, the learning system improves its performance. Theuser can train the device to provide optimal variable processing factorsfor different listening criteria, such as to maximize speechintelligibility, listening comfort, or pleasantness.

In one embodiment, based on the detected environment, the learningmachine can adjust amplification and gain control. It can optimize thegain at each or every frequency of the amplifier in a particularacoustic environment. The variable processing factor is can be theamplifier gain at each or every frequency. For example, gain at each orevery frequency of the amplifier can be a function of sound pressurelevel at the microphone in band i; SNRi=signal to noise ratio in band i;and SNRav=average SNR in all bands. Other acoustic or psychoacousticparameters, such as higher-order moments of the spectrum of the signaland variations of these moments with time, or other statisticalparameters or estimates of the acoustic signal or combinations of theseparameters, and optionally with other additional coefficients, can beused to calculate the gain or processing factors other than gain, suchas the speed at which the device reacts to change in the acousticenvironment. In another example, the learning machine can adjustparameters based on the environment by: (i) the volume of the outputsignal; (ii) the gain of the output signal at particular frequenciesrelative to other frequencies, for example the mid frequency gain can beboosted or attenuated with respect to the low or high band frequenciesof the output signal; and (iii) a slope control where the low and highfrequency band gains are adjusted in opposing directions while the midband gain is unchanged. In one embodiment, the learning machine candetect and adjust acoustic environment or psychoacoustic parameters thatare highly correlated to a particular listening criterion, such asmaximizing speech intelligibility or listening comfort. The variableprocessing factors may automatically adapt for the predicted weightedcombination of listening criteria as determined by the values of thecoefficients and the acoustic environment and/or psychoacousticparameters used by the learning system. The system further eventuallyallows the user to ‘train’ the device to adjust its operationautomatically. This has a number of advantages including: (1) greaterlevels of user satisfaction for both hearing aids and, because of thebetter control users have over their hearing; (2) improved listeningcomfort, and/or speech intelligibility and/or subjective sound quality,because processing parameters are optimized by each user to the personalpreference and needs of each user; (3) the ability to generalize the‘training’ algorithm to work effectively with future devices that havedifferent, or additional, parameters that require adjustment; (4)because the training is carried out by the user in daily life, expensiveclinical time is not required to achieve adjustment of the hearing aidand as adjustment is carried out in real life conditions, the adjustmentbetter relates to actual performance of the device, rather than idealclinical conditions; and (5) The ability to generalize the trainingalgorithm to work effectively in environments for which no training hasbeen provided by the user, by automatically applying the generalrelationships between preferred variable processing factors and acousticcharacteristics that have been established by the training.

FIG. 7A shows an exemplary learning machine such as a n artificialneural network (ANN) that can be trained on optimizing hearingassistance for the user. The ANN or connectionist systems are computingsystems vaguely inspired by the biological neural networks thatconstitute animal brains. Such systems “learn” to perform tasks byconsidering examples, generally without being programmed with anytask-specific rules. An ANN is based on a collection of connected unitsor nodes called artificial neurons, which loosely model the neurons in abiological brain. Each connection, like the synapses in a biologicalbrain, can transmit a signal from one artificial neuron to another. Anartificial neuron that receives a signal can process it and then signaladditional artificial neurons connected to it. Neural networks come inmany shapes and sizes and with varying degrees of complexity. Deepneural networks are defined as having at least two “hidden” processinglayers, which are not directly connected to a system's input or output.Each hidden layer refines the results fed to it by previous layers,adding in new considerations based on prior knowledge.

In one embodiment, machine learning is used for segregating sounds. Adigital filter that not only amplifies sound but can also isolate speechfrom background noise and automatically adjust the volumes of eachseparately. The system extract features that could distinguish voicesfrom noise based on common changes in amplitude, frequency, and themodulations of each. For example, the features can cover frequencies ofthe sounds and their intensities (loud or soft). The deep neural networkuses the features during an unsupervised training to distinguish speechfrom noise and adjusts the amplifier control settings. During operationand based on user responses under supervised training, the neuralnetwork learns from its success and failures and improves on theamplifier settings. This iteratively learning of noise in differentenvironments enables users to understand spoken words obscured by noise.Because it's not necessary for listeners to understand every word in aphrase to gather its meaning, this improvement frequently meant thedifference between comprehending a sentence or not.

The deep learning network can also be used to identify user health. Inembodiments that measure user health with heart rate, BI, ECG, EEG,temperature, or other health parameters, if an outlier situation exists,the system can flag to the user to follow up as an unusual sustainedvariation from normal health parameters. While this approach may notidentify exact causes of the variation, the user can seek help early.FIG. 7B shows an exemplary analysis with normal targets and outliers aswarning labels. For example, a patient may be mostly healthy, but whenhe or she is sick, the information pops out as outliers from the usualdata. Such outliers can be used to scrutinize and predict patienthealth. The data can be population based, namely that if a populationspatially or temporally has the same symptoms, and upon checking withthe medical hospitals or doctors to confirm the prediction, publichealth warnings can be generated. There are two main kinds of machinelearning techniques: Supervised learning: in this approach, a trainingdata sample with known relationships between variables is submittediteratively to the learning algorithm until quantitative evidence(“error convergence”) indicates that it was able to find a solutionwhich minimizes classification error. Several types of artificial neuralnetworks work according to this principle; and Unsupervised learning: inthis approach, the data sample is analyzed according to some statisticaltechnique, such as multivariate regression analysis, principalcomponents analysis, cluster analysis, etc., and automaticclassification of the data objects into subclasses might be achieved,without the need for a training data set.

Medical prognosis can be used to predict the future evolution of diseaseon the basis of data extracted from known cases such as the predictionof mortality of patients admitted to the Intensive Care Unit, usingphysiological and pathological variables collected at admission. Medicaldiagnosis can be done, where ML is used to learn the relationshipbetween several input variables (such as signs, symptoms, patienthistory, lab tests, images, etc.) and several output variables (thediagnosis categories). An example from my research: using symptomsrelated by patients with psychosis, an automatic classification systemwas devised to propose diagnoses of a particular disease. Medicaltherapeutic decisions can be done where ML is used to propose differenttherapies or patient management strategies, drugs, etc., for a givenhealth condition or diagnosis. Example from my research: patients withdifferent types of brain hematomas (internal bleeding) were used totrain a neural network so that a precise indication for surgery wasgiven after having learned the relationships between several inputvariables and the outcome. Signal or image analysis can be done, whereML is used to learn how features extracted from physiological signals(such as an EKG) or images (such as an x-ray, tomography, etc.) areassociated to some diagnoses. ML can even be used to extract featuresfrom signals or images, for example, in the so-called “signalsegmentation”. Example from my research: non-supervised algorithms wereused to extract different image textures from brain MRIs (magneticresonance imaging), such as bone, meninges, white matter, gray matter,vases, ventricles, etc., and then classifying automatically unknownimages, painting each identified region with a different color. Inanother example large data sets containing multiple variables obtainedfrom individuals in a given population (e.g., those living in acommunity, or who have a given health care plan, hospital, etc.), areused to train ML algorithms, so as to discover risk associations andpredictions (for instance, what patients have a higher risk of emergencyrisk readmissions or complications from diabetes. Public health canapply ML to predict, for instance, when and where epidemics are going tohappen in the future, such as food poisoning, infectious diseases, boutsof environmental diseases, and so on.

FIG. 7C shows an exemplary system to collect lifestyle and genetic datafrom various populations for subsequent prediction and recommendation tosimilarly situated users. The system collects attributes associated withindividuals that co-occur (i.e., co-associate, co-aggregate) withattributes of interest, such as specific disorders, behaviors andtraits. The system can identify combinations of attributes thatpredispose individuals toward having or developing specific disorders,behaviors and traits of interest, determining the level ofpredisposition of an individual towards such attributes, and revealingwhich attribute associations can be added or eliminated to effectivelymodify his or her lifestyle to avoid medical complications. Detailscaptured can be used for improving individualized diagnoses, choosingthe most effective therapeutic regimens, making beneficial lifestylechanges that prevent disease and promote health, and reducing associatedhealth care expenditures. It is also desirable to determine thosecombinations of attributes that promote certain behaviors and traitssuch as success in sports, music, school, leadership, career andrelationships. For example, the system captures information onepigenetic modifications that may be altered due to environmentalconditions, life experiences and aging. Along with a collection ofdiverse nongenetic attributes including physical, behavioral,situational and historical attributes, the system can predict apredisposition of a user toward developing a specific attribute ofinterest. In addition to genetic and epigenetic attributes, which can bereferred to collectively as pangenetic attributes, numerous otherattributes likely influence the development of traits and disorders.These other attributes, which can be referred to collectively asnon-pangenetic attributes, can be categorized individually as physical,behavioral, or situational attributes.

FIG. 7C displays one embodiment of the attribute categories and theirinterrelationships according to the one embodiment and illustrates thatphysical and behavioral attributes can be collectively equivalent to thebroadest classical definition of phenotype, while situational attributescan be equivalent to those typically classified as environmental. In oneembodiment, historical attributes can be viewed as a separate categorycontaining a mixture of genetic, epigenetic, physical, behavioral andsituational attributes that occurred in the past. Alternatively,historical attributes can be integrated within the genetic, epigenetic,physical, behavioral and situational categories provided they are madereadily distinguishable from those attributes that describe theindividual's current state. In one embodiment, the historical nature ofan attribute is accounted for via a time stamp or other time-basedmarker associated with the attribute. As such, there are no explicithistorical attributes, but through use of time stamping, the timeassociated with the attribute can be used to make a determination as towhether the attribute is occurring in what would be considered thepresent, or if it has occurred in the past. Traditional demographicfactors are typically a small subset of attributes derived from thephenotype and environmental categories and can be therefore representedwithin the physical, behavioral and situational categories.

Since the system captures information from various diverse populations,the data can be mined to discover combinations of attributes regardlessof number or type, in a population of any size, that causepredisposition to an attribute of interest. The ability to accuratelydetect predisposing attribute combinations naturally benefits from beingsupplied with datasets representing large numbers of individuals andhaving a large number and variety of attributes for each. Nevertheless,the one embodiment will function properly with a minimal number ofindividuals and attributes. One embodiment of the one embodiment can beused to detect not only attributes that have a direct (causal) effect onan attribute of interest, but also those attributes that do not have adirect effect such as instrumental variables (i.e., correlativeattributes), which are attributes that correlate with and can be used topredict predisposition for the attribute of interest but are not causal.For simplicity of terminology, both types of attributes are referred toherein as predisposing attributes, or simply attributes, that contributetoward predisposition toward the attribute of interest, regardless ofwhether the contribution or correlation is direct or indirect.

FIG. 7D shows a deep learning machine using deep convolutionary neuralnetworks for detecting genetic based drug-drug interaction. Oneembodiment uses an AlexNet: 8-layer architecture, while anotherembodiment uses a VGGNet: 16-layer architecture (each pooling layer andlast 2 FC layers are applied as feature vector). In one embodiment fordrugs, the indications of use and other drugs used capture most of manyimportant covariates. One embodiment access data from SIDER (atext-mined database of drug package inserts), the Offsides database thatcontains information complementary to that found in SIDER and improvesthe prediction of protein targets and drug indications, and the Twosidesdatabase of mined putative DDIs also lists predicted adverse events, allavailable at the http://PharmGKB.org Web site.

The system of FIG. 7D receives data on adverse events stronglyassociated with indications for which the indication and the adverseevent have a known causative relationship. A drug-event association issynthetic if it has a tight reporting correlation with the indication(p≥0.1) and a high relative reporting (RR) association score (RR≥2).Drugs reported frequently with these indications were 80.0 (95% CI, 14.2to 3132.8; P<0.0001, Fisher's exact test) times as likely to havesynthetic associations with indication events. Disease indications are asignificant source of synthetic associations. The moredisproportionately a drug is reported with an indication (x axis), themore likely that drug will be synthetically associated. For example,adverse events strongly associated with drugs are retrieved from thedrug's package insert. These drug-event pairs represent a set of knownstrong positive associations. Adverse events related to sex and race arealso analyzed. For example, for physiological reasons, certain eventspredominantly occur in males (for example, penile swelling andazoospermia). Drugs that are disproportionately reported as causingadverse events in males were more likely to be synthetically associatedwith these events. Similarly, adverse events that predominantly occur ineither relatively young or relatively old patients are analyzed.

“Off-label” adverse event data is also analyzed, and off-label usesrefer to any drug effect not already listed on the drug's packageinsert. For example, the SIDER database, extracted from drug packageinserts, lists 48,577 drug-event associations for 620 drugs and 1092adverse events that are also covered by the data mining. Offsidesrecovers 38.8% (18,842 drug-event associations) of SIDER associationsfrom the adverse event reports. Thus, Offsides finds differentassociations from those reported during clinical trials before drugapproval.

Polypharmacy side effects for pairs of drugs (Twosides) are alsoanalyzed. These associations are limited to only those that cannot beclearly attributed to either drug alone (that is, those associationscovered in Offsides). The database contains a significant associationfor which the drug pair has a higher side-effect association score,determined using the proportional reporting ratio (PRR), than those ofthe individual drugs alone. The system determines pairwise similaritymetrics between all drugs in the Offsides and SIDER databases. Thesystem can predict shared protein targets using drug-effectsimilarities. The side-effect similarity score between two drugs islinearly related to the number of targets that those drugs share.

The system can determine relationships between the proportion of sharedindications between a pair of drugs and the similarity of theirside-effect profiles in Offsides. The system can use side-effectprofiles to suggest new uses for old drugs. While the preferred systempredicts existing therapeutic indications of known drugs, the system canrecommend drug repurposing using drug-effect similarities in Offsides.Corroboration of class-wide interaction effects with EMRs. The systemcan identify DDIs shared by an entire drug class. The class-classinteraction analysis generates putative drug class interactions. Thesystem analyzes laboratory reports commonly recorded in EMRs that may beused as markers of these class-specific DDIs.

In one embodiment, the knowledge-based repository may aggregate relevantclinical and/or behavioral knowledge from one or more sources. In anembodiment, one or more clinical and/or behavioral experts may manuallyspecify the required knowledge. In another embodiment, an ontology-basedapproach may be used. For example, the knowledge-based repository mayleverage the semantic web using techniques, such as statisticalrelational learning (SRL). SRL may expand probabilistic reasoning tocomplex relational domains, such as the semantic web. The SRL mayachieve this using a combination of representational formalisms (e.g.,logic and/or frame based systems with probabilistic models). Forexample, the SRL may employ Bayesian logic or Markov logic. For example,if there are two objects—‘asian male’ and ‘smartness’, they may beconnected using the relationship ‘Asian males are smart’. Thisrelationship may be given a weight (e.g., 0.3). This relationship mayvary from time to time (populations trend over years/decades). Byleveraging the knowledge in the semantic web (e.g., all references anddiscussions on the web where ‘blonde’ and ‘smartness’ are used andassociated) the degree of relationship may be interpreted from thesentiment of such references (e.g., positive sentiment: TRUE; negativesentiment: FALSE). Such sentiments and the volume of discussions maythen be transformed into weights. Accordingly, although the systemoriginally assigned a weight of 0.3, based on information from semanticweb about Asian males and smartness, may be revised to 0.9.

In an embodiment, Markov logic may be applied to the semantic web usingtwo objects: first-order formulae and their weights. The formulae may beacquired based on the semantics of the semantic web languages. In oneembodiment, the SRL may acquire the weights based on probability valuesspecified in ontologies. In another embodiment, where the ontologiescontain individuals, the individuals can be used to learn weights bygenerative learning. In some embodiments, the SRL may learn the weightsby matching and analyzing a predefined corpus of relevant objects and/ortextual resources. These techniques may be used to not only to obtainfirst-order waited formulae for clinical parameters, but also generalinformation. This information may then be used when making inferences.

For example, if the first order logic is obesity causes hypertension,there are two objects involved: obesity and hypertension. If data onpatients with obesity and as to whether they were diagnosed withdiabetes or not is available, then the weights for this relationship maybe learnt from the data. This may be extended to non-clinical examplessuch as person's mood, beliefs etc.

The pattern recognizer may use the temporal dimension of data to learnrepresentations. The pattern recognizer may include a pattern storagesystem that exploits hierarchy and analytical abilities using ahierarchical network of nodes. The nodes may operate on the inputpatterns one at a time. For every input pattern, the node may provideone of three operations: 1. Storing patterns, 2. Learning transitionprobabilities, and 3. Context specific grouping.

A node may have a memory that stores patterns within the field of view.This memory may permanently store patterns and give each pattern adistinct label (e.g. a pattern number). Patterns that occur in the inputfield of view of the node may be compared with patterns that are alreadystored in the memory. If an identical pattern is not in the memory, thenthe input pattern may be added to the memory and given a distinctpattern number. The pattern number may be arbitrarily assigned and maynot reflect any properties of the pattern. In one embodiment, thepattern number may be encoded with one or more properties of thepattern.

In one embodiment, patterns may be stored in a node as rows of a matrix.In such an embodiment, C may represent a pattern memory matrix. In thepattern memory matrix, each row of C may be a different pattern. Thesedifferent patterns may be referred to as C-1, C-2, etc., depending onthe row in which the pattern is stored.

The nodes may construct and maintain a Markov graph. The Markov graphmay include vertices that correspond to the store patterns. Each vertexmay include a label of the pattern that it represents. As new patternsare added to the memory contents, the system may add new vertices to theMarkov graph. The system may also create a link between to vertices torepresent the number of transition events between the patternscorresponding to the vertices. For example, when an input pattern isfollowed by another input pattern j for the first time, a link may beintroduced between the vertices i and j and the number of transitionevents on that link may be set to 1. System may then increment thenumber of transition counts on the link from i and j whenever a patternfrom i to pattern j is observed. The system may normalize the Markovgraph such that the links estimate the probability of a transaction.Normalization may be achieved by dividing the number of transitionevents on the outgoing links of each vertex by the total number oftransition events from the vertex. This may be done for all vertices toobtain a normalized Markov graph. When normalization is completed, thesum of the transition probabilities for each node should add to 1. Thesystem may update the Markov graph continuously to reflect newprobability estimates.

The system may also perform context-specific grouping. To achieve this,the system may partition a set of vertices of the Markov graph into aset of temporal groups. Each temporal group may be a subset of that setof vertices of the Markov graph. The partitioning may be performed suchthat the vertices of the same temporal group are highly likely to followone another.

The node may use Hierarchical Clustering (HC) to for the temporalgroups. The HC algorithm may take a set of pattern labels and theirpair-wise similarity measurements as inputs to produce clusters ofpattern labels. The system may cluster the pattern labels such thatpatterns in the same cluster are similar to each other.

As data is fed into the pattern recognizer, the transition probabilitiesfor each pattern and pattern-of-patterns may be updated based on theMarkov graph. This may be achieved by updating the constructedtransition probability matrix. This may be done for each pattern inevery category of patterns. Those with higher probabilities may bechosen and placed in a separate column in the database called aprediction list.

Logical relationships among the patterns may be manually defined basedon the clinical relevance. This relationship is specified as first-orderlogic predicates along with probabilities. These probabilities may becalled beliefs. In one embodiment, a Bayesian Belief Network (BBN) maybe used to make predictions using these beliefs. The BBN may be used toobtain the probability of each occurrence. These logical relationshipsmay also be based on predicates stored the knowledge base.

The pattern recognizer may also perform optimization for thepredictions. In one embodiment, this may be accomplished by comparingthe predicted probability for a relationship with its actual occurrence.Then, the difference between the two may be calculated. This may be donefor p occurrences of the logic and fed into a K-means clusteringalgorithm to plot the Euclidean distance between the points. A centroidmay be obtained by the algorithm, forming the optimal increment to thedifference. This increment may then be added to the (p+1)th occurrence.Then, the process may be repeated. This may be done until the patternrecognizer predicts logical relationships up to a specified accuracythreshold. Then, the results may be considered optimal.

When a node is at the first level of the hierarchy, its input may comedirectly from the data source, or after some preprocessing. The input toa node at a higher-level may be the concatenation of the outputs of thenodes that are directly connected to it from a lower level. Patterns inhigher-level nodes may represent particular coincidences of their groupsof children. This input may be obtained as a probability distributionfunction (PDF). From this PDF, the probability that a particular groupis active may be calculated as the probability of the pattern that hasthe maximum likelihood among all the patterns belonging to that group.

The system can use an expert system that can assess hypertension inaccording with the guidelines. In addition, the expert system can usediagnostic information and apply the following rules to assesshypertension:

Hemoglobin/hematocrit: Assesses relationship of cells to fluid volume(viscosity) and may indicate risk factors such as hypercoagulability,anemia.

Blood urea nitrogen (BUN)/creatinine: Provides information about renalperfusion/function.

Glucose: Hyperglycemia (diabetes mellitus is a precipitator ofhypertension) may result from elevated catecholamine levels (increaseshypertension).

Serum potassium: Hypokalemia may indicate the presence of primaryaldosteronism (cause) or be a side effect of diuretic-therapy.

Serum calcium: Imbalance may contribute to hypertension.

Lipid panel (total lipids, high-density lipoprotein [HDL], low-densitylipoprotein [LDL], cholesterol, triglycerides, phospholipids): Elevatedlevel may indicate predisposition for/presence of atheromatous plaques.

Thyroid studies: Hyperthyroidism may lead or contribute tovasoconstriction and hypertension.

Serum/urine aldosterone level: May be done to assess for primaryaldosteronism (cause).

Urinalysis: May show blood, protein, or white blood cells; or glucosesuggests renal dysfunction and/or presence of diabetes.

Creatinine clearance: May be reduced, reflecting renal damage.

Urine vanillylmandelic acid (VMA) (catecholamine metabolite): Elevationmay indicate presence of pheochromocytoma (cause); 24-hour urine VMA maybe done for assessment of pheochromocytoma if hypertension isintermittent.

Uric acid: Hyperuricemia has been implicated as a risk factor for thedevelopment of hypertension.

Renin: Elevated in renovascular and malignant hypertension, salt-wastingdisorders.

Urine steroids: Elevation may indicate hyperadrenalism,pheochromocytoma, pituitary dysfunction, Cushing's syndrome.

Intravenous pyelogram (IVP): May identify cause of secondaryhypertension, e.g., renal parenchymal disease, renal/ureteral-calculi.

Kidney and renography nuclear scan: Evaluates renal status (TOD).

Excretory urography: May reveal renal atrophy, indicating chronic renaldisease.

Chest x-ray: May demonstrate obstructing calcification in valve areas;deposits in and/or notching of aorta; cardiac enlargement.

Computed tomography (CT) scan: Assesses for cerebral tumor, CVA, orencephalopathy or to rule out pheochromocytoma.

Electrocardiogram (ECG): May demonstrate enlarged heart, strainpatterns, conduction disturbances.

Note: Broad, notched P wave is one of the earliest signs of hypertensiveheart disease.

The system may also be adaptive. In one embodiment, every level has acapability to obtain feedback information from higher levels. Thisfeedback may inform about certain characteristics of informationtransmitted bottom-up through the network. Such a closed loop may beused to optimize each level's accuracy of inference as well as transmitmore relevant information from the next instance.

The system may learn and correct its operational efficiency over time.This process is known as the maturity process of the system. Thematurity process may include one or more of the following flow of steps:

a. Tracking patterns of input data and identifying predefined patterns(e.g. if the same pattern was observed several times earlier, thepattern would have already taken certain paths in the hierarchical nodestructure).

b. Scanning the possible data, other patterns (collectively called InputSets (IS)) required for those paths. It also may check for any feedbackthat has come from higher levels of hierarchy. This feedback may beeither positive or negative (e.g., the relevance of the informationtransmitted to the inferences at higher levels). Accordingly, the systemmay decide whether to send this pattern higher up the levels or not, andif so whether it should it send through a different path.c. Checking for frequently required ISs and pick the top ‘F’ percentileof them.d. Ensuring it keeps this data ready.

In one embodiment, information used at every node may act as agentsreporting on the status of a hierarchical network. These agents arereferred to as Information Entities (In En). In En may provide insightabout the respective inference operation, the input, and the resultwhich collectively is called knowledge.

This knowledge may be different from the KB. For example, the abovedescribed knowledge may include the dynamic creation of insights by thesystem based on its inference, whereas the KB may act as a reference forinference and/or analysis operations. The latter being an input toinference while the former is a product of inference. When thisknowledge is subscribed to by a consumer (e.g. administering system oranother node in a different layer) it is called “Knowledge-as-a-Service(KaaS)” One embodiment processes behavior models are classified intofour categories as follows:

a. Outcome-based;

b. Behavior-based;

c. Determinant-based; and

d. Intervention-based.

One or more of the following rules of thumb may be applied duringbehavioral modeling:

One or more interventions affect determinants;

One or more determinants affect behavior; and

One or more behaviors affect outcome.

A behavior is defined to be a characteristic of an individual or a grouptowards certain aspects of their life such as health, socialinteractions, etc. These characteristics are displayed as their attitudetowards such aspects. In analytical terms, a behavior can be consideredsimilar to a habit. Hence, a behavior may be observed POP′ for a givendata from a user. An example of a behavior is dietary habits.Determinants may include causal factors for behaviors. They either causesomeone to exhibit the same behavior or cause behavior change. Certaindeterminants are quantitative but most are qualitative. Examples includeone's perception about a food, their beliefs, their confidence levels,etc. Interventions are actions that affect determinants. Indirectly theyinfluence behaviors and hence outcomes. System may get both primary andsecondary sources of data. Primary sources may be directly reported bythe end-user and AU. Secondary data may be collected from sensors suchas their mobile phones, cameras, microphone, as well as those collectedfrom general sources such as the semantic web.

These data sources may inform the system about the respectiveinterventions. For example, to influence a determinant calledforgetfulness which relates to a behavior called medication, the systemsends a reminder at an appropriate time, as the intervention. Then,feedback is obtained whether the user took the medication or not. Thishelps the system in confirming if the intervention was effective.

The system may track a user's interactions and request feedback abouttheir experience through assessments. The system may use thisinformation as part of behavioral modeling to determine if the userinterface and the content delivery mechanism have a significant effecton behavior change with the user. The system may use this information tooptimize its user interface to make it more personalized over time tobest suit the users, as well as to best suit the desired outcome.

The system also may accommodate data obtained directly from theend-user, such as assessments, surveys, etc. This enables users to sharetheir views on interventions, their effectiveness, possible causes, etc.The system's understanding of the same aspects is obtained by way ofanalysis and service by the pattern recognizer.

Both system-perceived and end user-perceived measures of behavioralfactors may be used in a process called Perception Scoring (PS). In thisprocess, hybrid scores may be designed to accommodate both abovementioned aspects of behavioral factors. Belief is the measure ofconfidence the system has, when communicating or inferring oninformation. Initially higher beliefs may be set for user-perceivedmeasures.

Over time, as the system finds increasing patterns as well as obtainsfeedback in pattern recognizer, the system may evaluate theeffectiveness of intervention(s). If the system triggers an interventionbased on user-perceived measures and it doesn't have significant effecton the behavior change, the system may then start reducing its belieffor user-perceived measures and instead will increase its belief forsystem-perceived ones. In other words, the system starts believing lessin the user and starts believing more in itself. Eventually this reachesa stage where system can understand end-users and their behavioralhealth better than end-users themselves. When perception scoring is donefor each intervention, it may result in a score called InterventionEffectiveness Score (IES).

Perception scoring may be done for both end-users as well as AU. Suchscores may be included as part of behavior models during cause-effectanalysis.

Causes may be mapped with interventions, determinants, and behaviorrespectively in order of the relevance. Mapping causes withinterventions helps in back-tracking the respective AU for that cause.In simple terms, it may help in identifying whose actions have had apronounced effect on the end-user's outcome, by how much and using whichintervention. This is very useful in identifying AUs who are veryeffective with specific interventions as well as during certain eventcontext. Accordingly, they may be provided a score called AssociatedUser Influence Score. This encompasses information for a given end-user,considering all interventions and possible contexts relevant to theuser's case.

The system may construct one or plans including one or moreinterventions based on analysis performed, and may be implemented. Forexample, the system may analyze eligibility of an intervention for agiven scenario, evaluating eligibility of two or more interventionsbased on combinatorial effect, prioritizing interventions to be applied,based on occurrence of patterns (from pattern recognizer), and/orsubmitting an intervention plan to the user or doctor in a formatreadily usable for execution.

This system may rely on the cause-effect analysis for its planningoperations. A plan consists of interventions and a respectiveimplementation schedule. Every plan may have several versions based onthe users involved in it. For example, the system may have a separateversion for the physician as compared to a patient. They will in turn dothe task and report back to the system. This can be done either directlyor the system may indirectly find it based on whether a desired outcomewith the end user was observed or not.

The methodology may be predefined by an analyst. For every cause, whichcan be an intervention(s), determinant(s), behavior(s) or combinationsof the same, the analyst may specify one or more remedial actions. Thismay be specified from the causal perspective and not the contextualperspective. Accordingly, the system may send a variety of data andinformation to pattern recognizer and other services, as feedback, forthese services to understand about the users. This understanding mayaffect their next set of plans which in turn becomes an infinite cyclicsystem where system affects the users while getting affected by them atthe same time. Such a system is called a reflexive-feedback enabledsystem. The system may user both positive and negativereflexive-feedback, though the negative feedback aspect maypredominantly be used for identifying gaps that the system needs toaddress. The system may provide information, such as one or more newlyidentified patterns, to an analyst (e.g., clinical analyst or doctor).In the use case, the doctor may be presented with one or morenotifications to address the relationship between carbohydrates and themedication that the patient is taking. One embodiment of the systemoperation includes receiving feedback relating to the plan, and revisingthe plan based on the feedback; the feedback being one or more patientbehaviors that occur after the plan; the revised plan including one ormore additional interventions selected based on the feedback; the one ormore patient behaviors that occur after the plan include a behaviortransition; determining one or more persons to associate with theidentified intervention; automatically revising probabilities from thecollected information; storing the revised probabilities, wherein therevised probabilities are used to determine the plan; and/orautomatically make one or more inferences based on machine learningusing one or more of the clinical information, behavior information, orpersonal information. Hypertension metrics may be one type of metricsutilized within the principles of the present disclosure. A hypertensionscore can be based on any type of alpha-numeric or visual analog scale.Hypertension scales may or may not be clinically validated and may useany scale (e.g. 1-100, 1-10, 1-4), picture, symbol, color, character,number, sound, letter, or written description of hypertension tofacilitate the communication of a patient's hypertension level. The typeof hypertension scale used may be determined according to a patient'sand/or healthcare provider's preferences, and may also be determinedbased on the needs of a patient including, for example, the patient'sage and/or communication capability. In further embodiments, theselected hypertension scale(s) may be determined by a service provider,such as, e.g., an organization implementing the principles of thepresent disclosure via a suitable software program or application.

Another metric may include a functionality score. A functionality scorecan be based on any type of alpha-numeric or visual analog scale.Non-limiting examples include the American Chronic Pain AssociationQuality of Life (ACPA QoL) Scale, Global Assessment of Functioning (GAF)Scale, and Short Form SF-36 Health Survey. Functionality scales may ormay not be clinically validated and may use any picture, symbol, color,character, number, sound, letter, written description of quality oflife, or physical functioning to facilitate communication of a patient'sfunctionality level. The functionality score may be, e.g., based on anassessment of a patient's ability to exercise as well as perform dailytasks and/or perform routine tasks such as, e.g., getting dressed,grocery shopping, cooking, cleaning, climbing stairs, etc. In someembodiments, the selected functionality scale(s) may be determined by aservice provider, such as, e.g., an organization implementing theprinciples of the present disclosure via a suitable software program orapplication.

A further metric may include a patient's medication usage. Medicationuse encompasses pharmacologic and therapeutic agents used to treat,control, and/or alleviate hypertension, including prescription drugs aswell as over-the-counter medications, therapeutic agents, and othernon-prescription agents. Medication use may include different classes ofpharmacologic agents. Medication use can be reported in any appropriateunits, such as number of doses taken, percentage of treatment plancompleted, frequency of doses, and/or dose strength; and may alsospecify additional information such as the type of formulation taken andthe route of administration (oral, enteral, topical, transdermal,parenteral, sublingual etc.). Molecular alternatives (e.g., acid, salt,solvate, complex, and pro-drug forms, etc.) and formulations (e.g.,solid, liquid, powder, gel, and suspensions, etc.) are furthercontemplated. Reported medication use may, for example, include thenumber of doses and types of medication taken since a previous reportedmedication use, and may also indicate the number of closes and types ofmedication taken within a period of time, such as within, the previous 2hours, 4 hours, 6 hours, 12 hours, 18 hours, 24 hours, 36 hours, or 48hours. In some embodiments, for example, medication use may be reportedin terms of dosage units recommended by a manufacturer or healthcareprovider for a given medication (e.g., minimum, maximum, or range ofappropriate unit dosage per unit time).

Reported medication use may allow for tracking compliance with atreatment regime. For example, a record of reported medication use mayassist a healthcare provider in evaluating medication efficacy,adjusting dosage, and/or adding other medications as necessary.

In some embodiments of the present disclosure, a patient or healthcareprovider may create a patient profile comprising, e.g., identifying,characterizing, and/or medical information, including information abouta patient's medical history, profession, and/or lifestyle. Furtherexamples of information that may be stored in a patient profile includesdiagnostic information such as family medical history, medical symptoms,duration of hypertension, localized vs. general hypertension, etc.Further contemplated as part of a patient profile are non-pharmacologictreatment(s) (e.g., chiropractic, radiation, holistic, psychological,acupuncture, etc.), lifestyle characteristics (e.g., diet, alcoholintake, smoking habits), cognitive condition, behavioral health, andsocial well-being.

A patient profile may, for example, be stored in a database andaccessible for analysis of the patient's reported hypertension metrics.In some embodiments, a patient profile may be created before collectingand/or transmitting a set of hypertension metrics to be received by aserver and/or database in other embodiments, a patient profile may becreated concurrently with, or even after transmitting/receiving one ormore hypertension metrics. In some embodiments a patient profile may beused to establish one or more hypertension metric e and/or referencevalues. A patient profile may, for example, allow for setting thresholdvalues or ranges, wherein reported hypertension metrics that falloutside of those limits trigger an alert to be sent to the patient or ahealthcare provider. Threshold values, limits, or ranges may also be setwithout reference to a patient profile. In some embodiments, one or moretarget value(s) (e.g., hypertension metric value(s)) may be set todetermine how the reported hypertension metrics compare with the targetvalue(s).

The methods and systems disclosed herein may rely on one or morealgorithm(s) to analyze one or more of the described metrics. Thealgorithm(s) may comprise analysis of data reported in real-time, andmay also analyze data reported in real-time in conjunction withauxiliary data stored in a hypertension management database. Suchauxiliary data may comprise, for example, historical patient data suchas previously-reported hypertension metrics (e.g., hypertension scores,functionality scores, medication use), personal medical history, and/orfamily medical history. In some embodiments, for example, the auxiliarydata includes at least one set of hypertension metrics previouslyreported and stored for a patient. In some embodiments, the auxiliarydata includes a patient profile such as, e.g., the patient profiledescribed above. Auxiliary data may also include statistical data, suchas hypertension metrics pooled for a plurality of patients within asimilar group or subgroup. Further, auxiliary data may include clinicalguidelines such as guidelines relating to hypertension management,including evidence-based clinical practice guidelines on the managementof acute and/or chronic hypertension or other chronic conditions.

Analysis of a set of hypertension metrics according to the presentdisclosure may allow for calibration of the level, degree, and/orquality of hypertension experienced by providing greater context topatient-reported data. For example, associating a hypertension score of7 out of 10 with high functionality for a first patient, and the samescore with low functionality for a second patient may indicate arelatively greater debilitating effect of hypertension on the secondpatient than the first patient. Further, a high hypertension scorereported by a patient taking a particular medication such as opioidanalgesics may indicate a need to adjust the patient's treatment plan.Further, the methods and systems disclosed herein may provide a means ofassessing relative changes in a patient's distress due to hypertensionover time. For example, a hypertension score of 5 out of 10 for apatient who previously reported consistently lower hypertension scores,e.g., 1 out of 10, may indicate a serious issue requiring immediatemedical attention.

Any combination(s) of hypertension metrics may be used for analysis inthe systems and methods disclosed. In some embodiments, for example, theset of hypertension metrics comprises at least one hypertension scoreand at least one functionality score. In other embodiments, the set ofhypertension metrics may comprise at least one hypertension score, atleast one functionality score, and medication use. More than one set ofhypertension metrics may be reported and analyzed at a given time. Forexample, a first set of hypertension metrics recording a patient'scurrent status and a second set of hypertension metrics recording thepatient's status at an earlier time may both be analyzed and may also beused to generate one or more recommended actions.

Each hypertension metric may be given equal weight in the analysis, ormay also be given greater or less weight than other hypertension metricsincluded in the analysis. For example, a functionality score may begiven greater or less weight with respect to a hypertension score and/ormedication use. Whether and/or how to weigh a given hypertension metricmay be determined according to the characteristics or needs of aparticular patient. As an example, Patient A reports a hypertensionscore of 8 (on a scale of 1 to 10 where 10 is the most severehypertension) and a functionality score of 9 (on a scale of 1 to 10where 10 is highest functioning), while Patient B reports a hypertensionscore of 8 but a functionality score of 4. The present disclosureprovides for the collection, analysis, and reporting of thisinformation, taking into account the differential impact of onehypertension score on a patient's functionality versus that samehypertension score's impact on the functionality of a different patient.

Hypertension metrics may undergo a pre-analysis before inclusion in aset of hypertension metrics and subsequent application of one or morealgorithms. For example, a raw score may be converted or scaledaccording to one or more algorithm(s) developed for a particularpatient. In some embodiments, for example, a non-numerical raw score maybe converted to a numerical score or otherwise quantified prior to theapplication of one or more algorithms. Patients and healthcare providersmay retain access to raw data (e.g., hypertension metric data prior toany analysis)

Algorithm(s) according, to the present disclosure may analyze the set ofhypertension metrics according to any suitable methods known in the art.Analysis may comprise, for example, calculation of statistical averages,pattern recognition, application of mathematical models, factoranalysis, correlation, and/or regression analysis. Examples of analysesthat may be used herein include, but are not limited to, those disclosedin U.S. Patent Application Publication No. 2012/0246102 A1 the entiretyof which is incorporated herein by reference.

The present disclosure further provides for the determination of anaggregated hypertension assessment score. In some embodiments, forexample, a set of pairs metrics may be analyzed to generate acomprehensive and/or individualized assessment of hypertension bygenerating a composite or aggregated score. In such embodiments, theaggregated score may include a combination of at least one hypertensionscore, at least one functionality score, and medication use. Additionalmetrics may also be included in the aggregated score. Such metrics mayinclude, but are not limited to, exercise habits, mental well-being,depression, cognitive functioning, medication side effects, etc. Any ofthe aforementioned types of analyses may be used in determining anaggregated score.

The algorithm(s) may include a software program that may be availablefor download to an input device in various versions. In someembodiments, for example, the algorithm(s) may be directly downloadedthrough the Internet or other suitable communications means to providethe capability to troubleshoot a health issue in real-time. Thealgorithm(s) may also be periodically updated, e.g., provided contentchanges, and may also be made available for download to an input device.

The methods presently disclosed may provide a healthcare provider with amore complete record of a patient's day-to-day status. By having accessto a consistent data stream of hypertension metrics for a patient, ahealthcare provider may be able to provide the patient with timelyadvice and real-time coaching on hypertension management options andsolutions. A patient may, for example, seek and/or receive feedback onhypertension management without waiting for an upcoming appointment witha healthcare provider or scheduling a new appointment. Such real-timecommunication capability may be especially beneficial to providepatients with guidance and treatment options during intervals betweenappointments with a healthcare provider. Healthcare providers may alsobe able to monitor a patient's status between appointments to timelyinitiate, modify, or terminate a treatment plan as necessary. Forexample, a patient's reported medication use may convey whether thepatient is taking too little or too much medication. In someembodiments, an alert may be triggered to notify the patient and/or ahealthcare provider of the amount of medication taken, e.g., incomparison to a prescribed treatment plan. The healthcare providercould, for example, contact the patient to discuss the treatment plan.The methods disclosed herein may also provide a healthcare provider witha longitudinal review of how a patient responds to hypertension overtime. For example, a healthcare provider may be able to determinewhether a given treatment plan adequately addresses a patient's needsbased on review of the patient's reported hypertension metrics andanalysis thereof according to the present disclosure. Analysis ofpatient data according to the methods presently disclosed may generateone or more recommended actions that may be transmitted and displayed onan output device. In some embodiments, the analysis recommends that apatient make no changes to his/her treatment plan or routine. In otherembodiments, the analysis generates a recommendation that the patientseek further consultation with a healthcare provider and/or establishcompliance with a prescribed treatment plan. In other embodiments, theanalysis may encourage a patient to seek immediate medical attention.For example, the analysis may generate an alert to be transmitted to oneor more output devices, e.g., a first output device belonging to thepatient and a second output device belonging to a healthcare provider,indicating that the patient is in need of immediate medical treatment.In some embodiments, the analysis may not generate a recommended action.Other recommended actions consistent with the present disclosure may becontemplated and suitable according to the treatment plans, needs,and/or preferences for a given patient.

The present disclosure further provides a means for monitoring apatient's medication use to determine when his/her prescription will runout and require a refill. For example, a patient profile may be createdthat indicates a prescribed dosage and frequency of administration, aswell as total number of dosages provided in a single prescription. Asthe patient reports medication use, those hypertension metrics may betransmitted to a server and stored in a database in connection with thepatient profile. The patient profile stored on the database may thuscontinually update with each added metric and generate a notification toindicate when the prescription will run out based on the reportedmedication use. The notification may be transmitted and displayed on oneor more output devices, e.g., to a patient and/or one or more healthcareproviders. In some embodiments, the one or more healthcare providers mayinclude a pharmacist. For example, a pharmacist may receive notificationof the anticipated date a prescription will run out in order to ensurethat the prescription may be timely refilled.

Patient data can be input for analysis according to the systemsdisclosed herein through any data-enabled device including, but notlimited to, portable/mobile and stationary communication devices, andportable/mobile and stationary computing devices. Non-limiting examplesof input devices suitable for the systems disclosed herein include smartphones, cell phones, laptop computers, netbooks, personal computers(PCs), tablet PCs, fax machines, personal digital assistants, and/orpersonal medical devices. The user interface of the input device may beweb-based, such as a web page, or may also be a stand-alone application.Input devices may provide access to software applications via mobile andwireless platforms, and may also include web-based applications.

The input device may receive data by having a user, including, but notlimited to, a patient, family member, friend, guardian, representative,healthcare provider, and/or caregiver, enter particular information viaa user interface, such as by typing and/or speaking. In someembodiments, a server may send a request for particular information tobe entered by the user via an input device. For example, an input devicemay prompt a user to enter sequentially a set of hypertension metrics,e.g., a hypertension score, a functionality score, and informationregarding use of one or more medications (e.g., type of medication,dosage taken, time of day, route of administration, etc.). In otherembodiments, the user may enter data into the input device without firstreceiving a prompt. For example, the user may initiate an application orweb-based software program and select an option to enter one or morehypertension metrics. In some embodiments, one or more hypertensionscales and/or functionality scales may be preselected by the applicationor software program. For example, a user may have the option ofselecting the type of hypertension scale and/or functionality scale forreporting hypertension metrics within the application or softwareprogram. In other embodiments, an application or software program maynot include preselected hypertension scales or functionality scales suchthat a user can employ any hypertension scale and/or functionality scaleof choice.

The user interface of an input device may allow a user to associatehypertension metrics with a particular date and/or time of day. Forexample, a user may report one or more hypertension metrics to reflect apatient's present status. A user may also report one or morehypertension metrics to reflect a patient's status at an earlier time.

Patient data may be electronically transmitted from an input device overa wired or wireless medium to a server, e.g., a remote server. Theserver may provide access to a database for performing an analysis ofthe data transmitted, e.g., set of hypertension metrics. The databasemay comprise auxiliary data for use in the analysis as described above.In some embodiments, the analysis may be automated, and may also becapable of providing real-time feedback to patients and/or healthcareproviders.

The analysis may generate one or more recommended actions, and maytransmit the recommended action(s) over at wired or wireless medium fordisplay on at least one output device. The at least one output devicemay include, e.g., portable/mobile and stationary communication devices,and portable/mobile and stationary computing devices. Non-limitingexamples of output devices suitable for the systems disclosed hereininclude smart phones, cell phones, laptop computers, netbooks, personalcomputers (PCs), tablet PCs, fax machines, personal digital assistants,and/or personal medical devices. In some embodiments, the input deviceis the at least one output device. In other embodiments, the inputdevice is one of multiple output devices. In some embodiments of thepresent disclosure, the one or more recommended actions are transmittedand displayed on each of two output devices. In such an example, oneoutput device may belong to a patient and the other device may belong toa healthcare provider.

The present disclosure also contemplates methods and systems in alanguage suitable for communicating with the patient and/or healthcareprovider, including languages other than English. A patient's medicaldata may be subject to confidentiality regulations and protection.Transmitting, analyzing, and/or storing information according to themethods and systems disclosed herein may be accomplished through securemeans, including HIPPA-compliant procedures and use ofpassword-protected devices, servers, and databases.

The systems and methods presently disclosed may be especially beneficialin outpatient, home, and/or on-the-go settings. The systems and methodsdisclosed herein may also be used as an inpatient tool and/or incontrolled medication administration such as developing a personalizedtreatment plan. In addition to monitoring health parameters, the systemcan include interventional devices such as a defibrillator. Thedefibrillator function is enabled by providing electrical energy of aselected energy/power level/voltage/current level or intensity deliveredfor a selected duration upon sensing certain patterns of undesirableheart activity wherein said undesirable heart activity necessitates anexternal delivery of a controlled electrical energy pulse forstimulating a selected heart activity. The defibrillator function isenabled by an intelligent defibrillator appliance that operates in amanner similar to the functions of an intelligent ECG appliance with theadditional capability of providing external electrical stimuli via forexample a wireless contact system pasted on various locations of thetorso. The electrical stimuli are delivered in conjunction with theintelligent defibrillator device or the mobile device performing theadditional functions of an intelligent defibrillator appliance. Thecontrol actions for providing real time stimuli to the heart ofelectrical pulses, is enabled by the intelligent defibrillator applianceby itself or in conjunction with an external server/intelligentappliance where the protocols appropriate for the specific individualare resident. The defibrillation actions are controlled in conjunctionwith the real time ECG data for providing a comprehensive real-timesolution to the individual suffering from abnormal or life-threateningheart activity/myocardial infraction. Additionally, by continuouslywearing the paste on wireless contacts that can provide the electricalimpulse needed, the individual is instantaneously able to get real timeattention/action using a specifically designed wearable intelligentdefibrillator appliance or a combination of an intelligent ECG plusdefibrillator appliance. Further the mobile device such as a cellulartelephone or other wearable mobile devices can be configured with theappropriate power sources and the software for performing the additionalfunctions of an intelligent defibrillator appliance specificallytailored to the individual.

The cellular telephone/mobile device can receive signals from the ECGmachine/appliance or as an intermediary device that transmits/receivesthe ECG data and results from a stationary or portable ECG appliance.The ability of the individual to obtain an ECG profile of the heart at aselected time and in a selected location is critical to getting timelyattention and for survival. Getting attention within 10 to 20 minutes ofa heart attack is crucial beyond that the chances for survival diminishsignificantly. The smart phone helps the patient to quickly communicatehis/her location and or discover the location of the nearest health carefacility that has the requisite cardiac care facilities and otherfacilities. The mobile device that the individual is carrying on theperson is enabled to provide the exact location of the individual inconjunction with the global positioning system. In addition, the systemis enabled to provide the directions and estimated travel time to/fromthe health care facility to the specific mobile device/individual.

Yet other intervention can include music, image, or video. The music canbe synchronized with respect to a blood pulse rate in one embodiment,and in other embodiments to biorhythmic signal—either to match thebiorhythmic signal, or, if the signal is too fast or too slow, to goslightly slower or faster than the signal, respectively. In order toentrain the user's breathing, a basic melody is preferably played whichcan be easily identified by almost all users as corresponding to aparticular phase of respiration. On top of the basic melody, additionallayers are typically added to make the music more interesting, to theextent required by the current breathing rate, as described hereinabove.Typically, the basic melody corresponding to this breathing includesmusical cords, played continuously by the appropriate instrument duringeach phase. For some applications, it is desirable to elongate slightlythe length of one of the respiratory phases, typically, the expirationphase. For example, to achieve respiration which is 70% expiration and30% inspiration, a musical composition written for an E:I ratio of 2:1may be played, but the expiration phase is extended by asubstantially-unnoticed 16%, so as to produce the desired respirationtiming. The expiration phase is typically extended either by slowingdown the tempo of the notes therein, or by extending the durations ofsome or all of the notes.

Although music for entraining breathing is described hereinabove asincluding two phases, it will be appreciated by persons skilled in theart that the music may similarly include other numbers of phases, asappropriate. For example, user may be guided towards breathing accordingto a 1:2:1:3 pattern, corresponding to inspiration, breath holding(widely used in Yoga), expiration, and post-expiratory pause (reststate).

In one embodiment, the volume of one or more of the layers is modulatedresponsive to a respiration characteristic (e.g., inhalation depth, orforce), so as to direct the user to change the characteristic, or simplyto enhance the user's connection to the music by reflecting therein therespiration characteristic. Alternatively, or additionally, parametersof the sound by each of the musical instruments may be varied toincrease the user's enjoyment. For example, during slow breathing,people tend to prefer to hear sound patterns that have smootherstructures than during fast breathing and/or aerobic exercise. Furtheralternatively or additionally, random musical patterns and/or digitizednatural sounds (e.g., sounds of the ocean, rain, or wind) are added as adecoration layer, especially for applications which direct the user intovery slow breathing patterns. The inventor has found that during veryslow breathing, it is desirable to remove the user's focus from temporalstructures, particularly during expiration.

Still further alternatively or additionally, the server maintains amusical library, to enable the user to download appropriate music and/ormusic-generating patterns from the Internet into device. Often, as auser's health improves, the music protocols which were initially storedin the device are no longer optimal, so the user downloads the newprotocols, by means of which music is generated that is more suitablefor his new breathing training. The following can be done:

obtaining clinical data from one or more laboratory test equipment andchecking the data on a blockchain;

obtaining genetic clinical data from one or more genomic equipment andstoring genetic markers in the EMR/HER including germ line data andsomatic data over time;

obtaining clinical data from a primary care or a specialist physiciandatabase;

obtaining clinical data from an in-patient care database or from anemergency room database;

saving the clinical data into a clinical data repository;

obtaining health data from fitness devices or from mobile phones;

obtaining behavioral data from social network communications and mobiledevice usage patterns;

saving the health data and behavioral data into a health data repositoryseparate from the clinical data repository; and

providing a decision support system (DSS) to apply genetic clinical datato the subject, and in case of an adverse event for a drug or treatment,generating a drug safety signal to alert a doctor or a manufacturer,wherein the DSS includes rule-based alerts on pharmacogenetics, oncologydrug regimens, wherein the DSS performs ongoing monitoring of actionablegenetic variants.

FIG. 7E illustrates one embodiment of a system for collaborativelytreating a patient with a disease such as cancer. In this embodiment, atreating physician/doctor logs into a consultation system 1 andinitiates the process by clicking on “Create New Case” (500). Next, thesystem presents the doctor with a “New Case Wizard” which provides asimple, guided set of steps to allow the doctor to fill out an “InitialAssessment” form (501). The doctor may enter Patient or SubjectInformation (502), enter Initial Assessment of patient/case (504),upload Test Results, Subject Photographs and X-Rays (506), acceptPayment and Service Terms and Conditions (508), review Summary of Case(510), or submit Forms to a AI machine based “consultant” such as aHearing Service AI Provider (512). Other clinical information for thecancer subject includes the imaging or medical procedure directedtowards the specific disease that one of ordinary skill in the art canreadily identify. The list of appropriate sources of clinicalinformation for cancer includes but it is not limited to: CT scan, MRIscan, ultrasound scan, bone scan, PET Scan, bone marrow test, bariumX-ray, endoscopy, lymphangiogram, IVU (Intravenous urogram) or IVP (IVpyelogram), lumbar puncture, cystoscopy, immunological tests(anti-malignant antibody screen), and cancer marker tests.

After the case has been submitted, the AI Machine Consultant can loginto the system 1 and consult/process the case (520). Using the TreatingDoctors Initial Assessment and Photos/X-Rays, the Consultant will clickon “Case Consultation” to initiate the “Case Consultation Wizard” (522).The consultant can fill out the “Consultant Record Analysis” form (524).The consultant can also complete the “Prescription Form” (526) andsubmit completed forms to the original Treating Doctor (528). Once thecase forms have been completed by the Consulting Doctor, the TreatingDoctor can access the completed forms using the system. The TreatingDoctor can either accept the consultation results (i.e. a pre-filledPrescription form) or use an integrated messaging system to communicatewith the Consultant (530). The Treating Doctor can log into the system(532), click on Patient Name to review (534), review the ConsultationResults (Summary Letter and pre-filled Prescription Form) (536). Ifsatisfied, the Treating Doctor can click “Approve Treatment” (538), andthis will mark the case as having being approved (540). The TreatingDoctor will be able to print a copy of the Prescription Form and theSummary Letter for submission to hearing aid manufacturer or provider(542). Alternatively, if not satisfied, the Treating Doctor can initiatea computer dialog with the Consultant by clicking “Send a Message”(544). The Treating Doctor will be presented with the “Send a Message”screen where a message about the case under consultation can be written(546). After writing a message, the Treating Doctor would click “Submit”to send the message to the appropriate Consultant (548). The Consultantwill then be able to reply to the Treating Doctor's Message and send amessage/reply back to the Treating Doctor (550).

Blockchain Authentication

Since the ITE sensors are IoT machines, the ITE device can negotiatecontracts on their own (without human) and exchange items of value bypresenting an open transaction on the associated funds in theirrespective wallets. Blockchain token ownership is immediatelytransferred to a new owner after authentication and verification, whichare based on network ledgers within a peer-to-peer network, guaranteeingnearly instantaneous execution and settlement.

A similar process is used to provide secure communications between IoTdevices, which is useful for edge IoT devices. The industrial world isadding billions of new IoT devices and collectively these devicesgenerate many petabytes of data each day. Sending all of this data tothe cloud is not only very cost prohibitive but it also creates agreater security risk. Operating at the edge ensures much fasterresponse times, reduced risks, and lower overall costs. Maintainingclose proximity to the edge devices rather than sending all data to adistant centralized cloud, minimizes latency allowing for maximumperformance, faster response times, and more effective maintenance andoperational strategies. In addition to being highly secure, the systemalso significantly reduces overall bandwidth requirements and the costof managing widely distributed networks.

In some embodiments, the described technology provides a peer-to-peercryptographic currency trading method for initiating a market exchangeof one or more Blockchain tokens in a virtual wallet for purchasing anasset (e.g., a security) at a purchase price. The system can determine,via a two-phase commit, whether the virtual wallet has a sufficientquantity of Blockchain tokens to purchase virtual assets (such aselectricity only from renewable solar/wind/ . . . sources, weather dataor location data) and physical asset (such as gasoline for automatedvehicles) at the purchase price. In various embodiments, in response toverifying via the two-phase commit that the virtual wallet has asufficient quantity of Blockchain tokens, the IoT machine purchases (orinitiates a process in furtherance of purchasing) the asset with atleast one of the Blockchain tokens. In one or more embodiments, if thedescribed technology determines that the virtual wallet has insufficientBlockchain tokens for purchasing the asset, the purchase is terminatedwithout exchanging Blockchain tokens.

The present system provides smart contract management with modules thatautomates the entire lifecycle of a legally enforceable smart contractby providing tools to author the contract so that it is bothjudge/arbitrator/lawyer readable and machine readable, and ensuring thatall contractual obligations are met by integrating with appropriateexecution systems, including traditional court system, arbitrationsystem, or on-line enforcement system. Different from theblockchain/bitcoin contract system where payment is made in advance andreleased when the conditions are electronically determined to besatisfied, this embodiment creates smart contracts that are verifiable,trustworthy, yet does not require advance payments that restrict theapplicability of smart contracts. The system has a contract managementsystem (CMS) that helps users in creating smart contracts fordeployment. After template creation, FIG. 13A shows a flow diagram ofthe functionality of system in accordance with one embodiment whenauthoring a contract using one of the smart contract templates. In oneembodiment, the functionality of the flow diagram of FIG. 13A isimplemented by software stored in memory and executed by a processor. Inother embodiments, the functionality can be performed by hardware, orany combination of hardware and software.

A smart contract is a computerized transaction protocol that executesthe terms of a contract. A smart contract can have the following fields:object of agreement, first party blockchain address, second partyblockchain address, essential content of contract, signature slots andblockchain ID associated with the contract. Turning now to FIG. 13A, at2, the user logs into the system to author a smart contract. The systemthen retrieves the appropriate contract template for the user, and auser interface renderer displays the corresponding deal sheet userinterface to the user. The selection of the appropriate contracttemplate can be based on many factors, including the role of the user,the intended parties to the contract, the type of contract desired, etc.At 4, the user enters the information that is requested by the userinterface based on the attributes displayed. Because the user interfaceis tailored specifically to the desired type of contract, the requiredcontract terms information for that type of contract will be entered bythe user as guided by the attributes of the template. The user mayinteract with the user interface through a single page or throughmultiple pages in a particular sequence with a forms wizard, or throughthe selection of tabs. In one embodiment, the user interface is renderedas a mark-up language such as XML showing the structure of therequirements of the contract. In other embodiments, the user interfaceis rendered as an Excel worksheet, Word document, or other applicationcompatible format that can be read by the contracting parties, lawyers,judges, and jury. At 6, the contract is generated based on the userinput to the user interface. The contract can be in the form ofbytecodes for machine interpretation or can be the markup language forhuman consumption. If there are other contracts that are incorporated byreference, the other contracts are formed in a nested hierarchy similarto program language procedures/subroutines and then embedded inside thecontract. At 8, the smart contract is assigned a unique block chainnumber and inserted into block chain. At 10, the smart contract is sentto one or more recipients, which open the payload and execute the termsof the contract and if specified contractual conditions are met, thesmart contract can authorize payment. At 12, if dispute arise, the CMScan graphically decode the contract terms in the smart contract for ajudge, jury, or lawyer to apply legal analysis and determine theparties' obligations.

Cloud Storage Security

In another aspect, a distributed file storage system includes nodes areincentivized to store as much of the entire network's data as they can.Blockchain currency is awarded for storing files, and is transferred inBitcoin or Ether transactions, as in. Files are added to the network byspending currency. This produces strong monetary incentives forindividuals to join and work for the network. In the course of ordinaryoperation of the storage network, nodes contribute useful work in theform of storage and distribution of valuable data.

In another aspect, a method for providing electronic content retrievalwith cloud computing is provided. A first request message is received inreal-time on the first cloud application stored on the cloud servernetwork device with the one or more processors from a second cloudapplication. The first request message includes a request for desiredcloud electronic content stored in the plural cloud storage objectsstored on the selected ones of the plural other different cloud servernetwork devices located on one or more of the networks comprising thecloud communications network. The plural different cloud storage objectsfunction as a single secure storage object for electronic content on thecloud communications network. A cloud content location map is retrievedsecurely on the first cloud application on the cloud server networkdevice. The cloud content location map includes address locations of theselected ones of the plural other different cloud server network deviceson the cloud communications network. The first cloud application on thecloud server network device sends plural second request messages for thedesired cloud electronic content to the selected ones of the pluralother different cloud server network devices identified in the retrievedcloud content location map and located on one or more of the publiccommunication networks, the one or more private networks, communitynetworks and hybrid networks comprising the cloud communicationsnetwork. The first cloud application on the first server network devicecombines the one or more individual components of the desired cloudelectronic content from the plural cloud storage objects from thereceived plural response messages into a final desired electronic cloudcontent component. The first cloud application on the cloud servernetwork device securely sends in real-time the final desired cloudelectronic content component as the request desired cloud electroniccontent to the target network device via the cloud communicationsnetwork. The second cloud application on the target network devicecannot determine the desired cloud electronic content was split and wasstored in plural cloud storage objects and cannot determine which ofplural selected ones of the other different cloud server network deviceson which ones of the public, private, community or hybrid networks onthe cloud communications network may have stored portions of the finaldesired cloud electronic content, thereby providing a second and/orfourth layer of security and privacy for the desired cloud electroniccontent on the cloud communications network.

To enable an IOT device such as a car or a robot to access cloud datasecurely, and to grant access right to agents of the IOT device such asmedia players in the car, for example, the following methods can be usedfor accessing data, content, or application stored in a cloud storage,comprising: authorizing a first client device; receiving anauthorization request from the first client device; generating anauthorization key for accessing the cloud server and storing the key ina blockchain; providing the authorization key to the first clientdevice; receiving the authorization key from an IOT device as a secondclient device working as an agent of the first client device; grantingaccess to the second client device based on the authorization key;receiving a map of storage locations of cloud objects associated with anapplication or content, each storage location identified in ablockchain; and reassembling the application or content from the storagelocations.

In implementation, the blockchain is decentralized and does not requirea central authority for creation, processing or verification andcomprises a public digital ledger of all transactions that have everbeen executed on the blockchain and wherein new blocks are added to theblockchain in a linear, chronological order. The public digital ledgerof the blockchain comprises transactions and blocks. Blocks in theblockchain record and confirm when and in what sequence transactions areentered and logged into the blockchain. The transactions comprisedesired electronic content stored in the blockchain. The desiredelectronic content includes a financial transaction. The financialtransaction includes a cryptocurrency transaction, wherein thecryptocurrency transaction includes a BITCOIN or an ETHEREUMtransaction. An identifier for the received one or more blocks in theblockchain includes a private encryption key.

Medical History

The above permissioned blockchain can be used to share sensitive medicaldata with different authorized institutions. The institutions aretrusted parties and vouched for by the trusted pont. A Patient-ProviderRelationship (PPR) Smart Contract is issued when one node from a trustedinstitution stores and manages medical records for the patient. The PPRdefines an assortment of data pointers and associated access permissionsthat identify the records held by the care provider. Each pointerconsists of a query string that, when executed on the provider'sdatabase, returns a subset of patient data. The query string is affixedwith the hash of this data subset, to guarantee that data have not beenaltered at the source. Additional information indicates where theprovider's database can be accessed in the network, i.e. hostname andport in a standard network topology. The data queries and theirassociated information are crafted by the care provider and modifiedwhen new records are added. To enable patients to share records withothers, a dictionary implementation (hash table) maps viewers' addressesto a list of additional query strings. Each string can specify a portionof the patient's data to which the third party viewer is allowed access.For SQL data queries, a provider references the patient's data with aSELECT query on the patient's address. For patients uses an interfacethat allows them to check off fields they wish to share through agraphical interface. The system formulates the appropriate SQL queriesand uploads them to the PPR on the blockchain.

In one embodiment, the transaction 303 includes the recipient's address324 (e.g., a hash value based on the receiver's public key), theBlockchain token 309 (i.e., a patient ID 328 and personally identifiableinformation such as Social Security 326), past medical institutionrelationship information 331 (if any), and optional other information310. The transaction 323 is digitally signed by the patient who is thesender's private key to create a digital signature 332 for verifying thesender's identity to the network nodes. The network nodes decrypt thedigital signature 332, via the sender's previously exchanged public key,and compare the unencrypted information to the transaction 323. If theymatch, the sender's authenticity is verified and, after a proper chainof ownership is verified via the ledgers (as explained above), thereceiver is recorded in the ledgers as the new Blockchain token 329authorized owner of the medical information. Block 328 of FIG. 13G canpoint to off-chain storage warehouses containing the patient's medicalhistory so that the current owner (or all prior owners) can access thepatient medical information for treatment. Further, the information canbe segmented according to need. This way, if a medication such ascannabis that requires the patient to be an adult, the system can bequeried only to the information needed (such as is this patient anadult) and the system can respond only as to the query and there is noneed to send other question (in the adult age example, the systemreplies only adult or not and does not send the birthday to theinquiring system).

In another embodiment, the system includes two look up tables, a globalregistration look up table (GRLT) where all participants (medicalinstitutions and patients) are recorded with name or identity string,blockchain address for the smart contract, and Patient-Provider lookuptable (PPLT). This is maintained by a trusted host authority such as agovernment health authority or a government payor authority. Oneembodiment maps participant identification strings to their blockchainaddress or Ethereum address identity (equivalent to a public key). Termsin the smart contract can regulate registering new identities orchanging the mapping of existing ones. Identity registration can thus berestricted only to certified institutions. The PPLT maps identitystrings to an address on the blockchain. Patients can poll their PPLTand be notified whenever a new relationship is suggested or an update isavailable. Patients can accept, reject or delete relationships, decidingwhich records in their history they acknowledge. The accepting orrejecting relationships is done only by the patients. To avoidnotification spamming from malicious participants, only trustedproviders can update the status variable. Other contract terms or rulescan specify additional verifications to confirm proper actor behavior.

When Provider 1 adds a record for a new patient, using the GRLT on theblockchain, the patient's identifying information is first resolved totheir matching Ethereum address and the corresponding PPLT is located.Provider 1 uses a cached GRLT table to look up any existing records ofthe patient in the PPLT. For all matching PPLTs, Provider 1 broadcasts asmart contract requesting patient information to all matching PPLTentries. If the cache did not produce a result for the patient identitystring or blockchain address, Provider 1 can send a broadcast requestinginstitutions who handles the patient identity string or the blockchainaddress to all providers. Eventually, Provider 2 responds with itsaddresses. Provider 2 may insert an entry for Provider 1 into itsaddress resolution table for future use. Provider 1 caches the responseinformation in its table and can now pull information from Provider 2and/or supplement the information known to Provider 2 with hashedaddresses to storage areas controlled by Provider 1.

Next, the provider uploads a new PPR to the blockchain, indicating theirstewardship of the data owned by the patient's Ethereum address. Theprovider node then crafts a query to reference this data and updates thePPR accordingly. Finally, the node sends a transaction which links thenew PPR to the patient's PPLT, allowing the patient node to later locateit on the blockchain.

A Database Gatekeeper provides an off-chain, access interface to thetrusted provider node's local database, governed by permissions storedon the blockchain. The Gatekeeper runs a server listening to queryrequests from clients on the network. A request contains a query string,as well as a reference to the blockchain PPR that warrants permissionsto run it. The request is cryptographically signed by the issuer,allowing the gatekeeper to confirm identities. Once the issuer'ssignature is certified, the gatekeeper checks the blockchain contractsto verify if the address issuing the request is allowed access to thequery. If the address checks out, it runs the query on the node's localdatabase and returns the result over to the client.

A patient selects data to share and updates the corresponding PPR withthe third-party address and query string. If necessary, the patient'snode can resolve the third party address using the GRLT on theblockchain. Then, the patient node links their existing PPR with thecare provider to the third-party's Summary Contract. The third party isautomatically notified of new permissions, and can follow the link todiscover all information needed for retrieval. The provider's DatabaseGatekeeper will permit access to such a request, corroborating that itwas issued by the patient on the PPR they share.

In one embodiment that handles persons without previous blockchainhistory, admitting procedures are performed where the person's personaldata is recorded and entered into the blockchain system. This data mayinclude: name, address, home and work telephone number, date of birth,place of employment, occupation, emergency contact information,insurance coverage, reason for hospitalization, allergies to medicationsor foods, and religious preference, including whether or not one wishesa clergy member to visit, among others. Additional information mayinclude past hospitalizations and surgeries, advance directives such asa living will and a durable power to attorney. During the time spent inadmitting, a plastic bracelet will be placed on the person's wrist withtheir name, age, date of birth, room number, and blockchain medicalrecord reference on it.

The above system can be used to connect the blockchain with differentEHR systems at each point of care setting. Any time a patient isregistered into a point of care setting, the EHR system sends a messageto the GRLT to identify the patient if possible. In our example, PatientA is in registration at a particular hospital. The PPLT is used toidentify Patient A as belonging to a particular plan. The smartcontracts in the blockchain automatically updates Patient A's care plan.The blockchain adds a recommendation to put Patient A by looking at thecomplete history of treatments by all providers and optimizes treat. Forexample, the system can recommend the patient be enrolled in a weightloss program after noticing that the patient was treated for sedentarylifestyle, had history of hypertension, and the family history indicatesa potential heart problem. The blockchain data can be used forpredictive analytics, allowing patients to learn from their familyhistories, past care and conditions to better prepare for healthcareneeds in the future. Machine learning and data analysis layers can beadded to repositories of healthcare data to enable a true “learninghealth system” can support an additional analytics layer for diseasesurveillance and epidemiological monitoring, physician alerts ifpatients repeatedly fill and abuse prescription access.

In one embodiment, an IOT medical device captures patient data in thehospital and automatically communicates data to a hospital database thatcan be shared with other institutions or doctors. First, the patient IDand blockchain address is retrieved from the patient's wallet and themedical device attaches the blockchain address in a field, along withother fields receiving patient data. Patient data is then stored in ahospital database marked with the blockchain address and annotated by amedical professional with interpretative notes. The notes are affiliatedwith the medical professional's blockchain address and the PPR blockchain address. A professional can also set up the contract termsdefining a workflow. For example, if the device is a blood pressuredevice, the smart contract can have terms that specify dietaryrestrictions if the patient is diabetic and the blood pressure isborderline and food dispensing machines only show items with low saltand low calorie, for example.

The transaction data may consist of a Colored Coin implementation(described in more detail at https://en.bitcoin.it/wiki/Colored_Coinswhich is incorporated herein by reference), based on Open Assets(described in more detail athttps://github.com/OpenAssets/open-assets-protocol/blob/master/specification.mediawikiwhich is incorporated herein by reference), using on the OP_RETURNoperator. Metadata is linked from the Blockchain and stored on the web,dereferenced by resource identifiers and distributed on public torrentfiles. The colored coin specification provides a method fordecentralized management of digital assets and smart contracts(described in more detail athttps://github.com/ethereum/wiki/wiki/White-Paper which is incorporatedherein by reference.) For our purposes the smart contract is defined asan event-driven computer program, with state, that runs on a blockchainand can manipulate assets on the blockchain. So a smart contract isimplemented in the blockchain scripting language in order to enforce(validate inputs) the terms (script code) of the contract.

Patient Behavior and Risk Pool Rated Health Plans

With the advent of personal health trackers, new health plans arerewarding consumers for taking an active part in their wellness. Thesystem facilitates open distribution of the consumers wellness data andprotect it as PHR must be, and therefore prevent lock-in of consumers,providers and payers to a particular device technology or health plan.In particular, since PHR data is managed on the blockchain a consumerand/or company can grant access to a payer to this data such that thepayer can perform group analysis of an individual or an entire company'semployee base including individual wellness data and generate a riskscore of the individual and/or organization. Having this information,payers can then bid on insurance plans tailored for the specificorganization. Enrollment then, also being managed on the blockchain, canbecome a real-time arbitrage process. The pseudo code for the smartcontract to implement a patient behavior based health plan is asfollows.

store mobile fitness data

store consumer data in keys with phr_info, claim_info, enrollment_info

for each consumer:

add up all calculated risk for the consumer

determine risk score based on mobile fitness data

update health plan cost based on patient behavior

Patient and Provider Data Sharing

A patient's Health BlockChain wallet stores all assets, which in turnstore reference ids to the actual data, whether clinical documents inHL7 or FHIR format, wellness metrics of activity and sleep patterns, orclaims and enrollment information. These assets and control of grants ofaccess to them is afforded to the patient alone. A participatingprovider can be given full or partial access to the data instantaneouslyand automatically via enforceable restrictions on smart contracts.

Utilizing the Health BlockChain the access to a patient's PHR can begranted as part of scheduling an appointment, during a referraltransaction or upon arrival for the visit. And, access can just aseasily be removed, all under control of the patient.

Upon arrival at the doctor's office, an application automatically logsinto a trusted provider's wireless network. The app is configured toautomatically notify the provider's office of arrival and grant accessto the patient's PHR. At this point the attending physician will haveaccess to the patient's entire health history. The pseudo code for thesmart contract to implement a patient and provider data sharing is asfollows.

Patient download apps and provide login credential and logs into theprovider wireless network

Patient verifies that the provider wireless network belongs to a patienttrusted provider list

Upon entering provider premise, system automatically logs in and grantsaccess to provider

Patient check in data is automatically communicated with provider systemto provide PHR

Provider system synchronizes files and obtain new updates to the patientPHR and flags changes to provider.

Patient Data Sharing

Patient's PHR data is valuable information for their personal healthprofile in order to provide Providers (Physicians) the necessaryinformation for optimal health care delivery. In addition this clinicaldata is also valuable in an aggregate scenario of clinical studies wherethis information is analyzed for diagnosis, treatment and outcome.Currently this information is difficult to obtain due to the siloedstorage of the information and the difficulty on obtaining patientpermissions.

Given a patient Health BlockChain wallet that stores all assets asreference ids to the actual data. These assets can be included in anautomated smart contract for clinical study participation or any otherdata sharing agreement allowed by the patient. The assets can be sharedas an instance share by adding to the document a randomized identifieror nonce, similar to a one-time use watermark or serial number, a uniqueasset (derived from the original source) is then generated for aparticular access request and included in a smart contract as an inputfor a particular request for the patient's health record information. Apatient can specify their acceptable terms to the smart contractregarding payment for access to PHR, timeframes for acceptable access,type of PHR data to share, length of history willing to be shared,de-identification thresholds or preferences, specific attributes of theconsumer of the data regarding trusted attributes such as reputation,affiliation, purpose, or any other constraints required by the patient.Attributes of the patient's data are also advertised and summarized asproperties of the smart contract regarding the type of diagnosis andtreatments available. Once the patient has advertised their willingnessto share data under certain conditions specified by the smart contractit can automatically be satisfied by any consumer satisfying the termsof the patient and their relevance to the type of PHR needed resultingin a automated, efficient and distributed means for clinical studies toconsume relevant PHR for analysis. This process provides an automatedexecution over the Health BlockChain for any desired time period thatwill terminate at an acceptable statistical outcome of the requiredattained significance level or financial limit. The pseudo code for thesmart contract to implement automated patient data sharing is asfollows.

Patient download apps and provide login credential and logs into theclinical trial provider wireless network

Patient verifies that the provider wireless network belongs to a patienttrusted provider list

Upon entering provider premise, system automatically logs in and grantsaccess to provider

Patient check in data is automatically communicated with provider systemto provide clinical trial data

In one embodiment, a blockchain entry is added for each touchpoint ofthe medication as it goes through the supply chain from manufacturingwhere the prescription package serialized numerical identification (SNI)is sent to wholesalers who scan and record the SNI and location and thento distributors, repackagers, and pharmacies, where the SNI/locationdata is recorded at each touchpoint and put on the blockchain. Themedication can be scanned individually, or alternatively can be scannedin bulk. Further, for bulk shipments with temperature and shock sensorsfor the bulk package, temperature/shock data is captured with theshipment or storage of the medication.

A smart contract assesses against product supply chain rule and cancause automated acceptance or rejection as the medication goes througheach supply chain touchpoint. The process includes identifying aprescription drugs by query of a database system authorized to track andtrace prescription drugs or similar means for the purpose of monitoringthe movements and sale of pharmaceutical products through a supplychain; a.k.a. e-pedigree trail; serialized numerical identification(SNI), stock keeping units (SKU), point of sale system (POS), systemsetc. in order to compare the information; e.g. drug name, manufacturer,etc. to the drug identified by the track and trace system and to ensurethat it is the same drug and manufacturer of origin. The process canverify authenticity and check pedigree which can be conducted at anypoint along the prescription drug supply chain; e.g. wholesaler,distributor, doctor's office, pharmacy. The most optimal point forexecution of this process would be where regulatory authorities view thegreatest vulnerability to the supply chain's integrity. For example,this examination process could occur in pharmacy operations prior tocontainerization and distribution to the pharmacy for dispensing topatients.

An authenticated prescription drug with verified drug pedigree trail canbe used to render an informational object, which for the purpose ofillustration will be represented but not be limited to a unique mark;e.g. QR Code, Barcode, Watermark, Stealth Dots, Seal or 2 Dimensionalgraphical symbol, hereinafter called a certificate, seal, or mark. Anexemplary embodiment for use of said certificate, mark, or seal can beused by authorized entities as a warrant of the prescription drug'sauthenticity and pedigree. For example, when this seal is appended to aprescription vial presented to a patient by a licensed pharmacy, itwould represent the prescription drug has gone through an authenticationand logistics validation process authorized by a regulatory agency (s);e.g. HHS, FDA, NABP, VIPP, etc. An exemplary embodiment for use of saidcertificate, mark or seal would be analogous to that of the functioningfeatures, marks, seals, and distinguishing characteristics thatcurrently authenticate paper money and further make it difficult tocounterfeit. Furthermore, authorized agents utilizing the certificateprocess would be analogous to banks participating in the FDIC program.

A user; e.g. patient equipped with the appropriate application on aportable or handheld device can scan the certificate, mark or seal andreceive an audible and visible confirmation of the prescription drug'sname, and manufacturer. This will constitute a confirmation of theauthenticity of the dispensed prescription drug. Extensible use of thecertificate, mark, or seal will include but not be limited to; gainingaccess to website (s) where additional information or interactivefunctions can be performed; e.g. audible narration of the drug'scharacteristics and physical property descriptions, dosing, information,and publications, etc. A user; e.g. patient equipped with theappropriate application on a portable or handheld device can scan thecertificate, mark, or seal and be provided with notifications regarding;e.g. immediate recall of the medication, adverse events, newformulations, critical warnings of an immediate and emergency naturemade by prescription drug regulatory authorities and, or their agents. Auser; e.g. patient equipped with a portable or handheld device with theappropriate application software can use the portable and, or handhelddevice to store prescription drug information in a secure, non-editableformat on their device for personal use; e.g. MD's Office Visits,Records Management, Future Authentications, Emergency use by firstresponders etc. A user; e.g. patient equipped with the appropriateapplication on a portable or handheld device can scan the drug via anoptical scan, picture capture, spectroscopy or other means ofidentifying its physical properties and characteristics; e.g. spectralsignature, size, shape, color, texture, opacity, etc and use this datato identify the prescription drug's name, and manufacturer. A user; e.g.patient equipped with the appropriate application on a portable orhandheld device and having the certification system can receive updatedinformation (as a subscriber in a client/server relationship) on acontinuing or as needed ad hoc basis (as permitted) about notificationsmade by prescription drug regulatory authorities regarding; e.g.immediate recall of medications, adverse events, new formulations andcritical warnings of an immediate and emergency nature. A user; e.g.patient, subscriber to the certificate system equipped with theappropriate application on a portable or handheld device will benotified by audible and visible warnings of potential adverse affectsbetween drug combinations stored in their device's memory of previously“Certified Drugs.” A user; e.g. patient subscriber to the certificationsystem equipped with the appropriate application on a portable orhandheld device will receive notification of potential adverse affectsfrom drug combinations, as reported and published by medicalprofessionals in documents and databases reported to; e.g. DrugEnforcement Administration (DEA), Health and Human Services, (HHS) Foodand Drug Administration, (FDA) National Library of Medicines, (NLM) andtheir agents; e.g., Daily Med, Pillbox, RX Scan, PDR, etc.

1. A method for prescription drug authentication by receiving acertificate representing manufacturing origin and distributiontouchpoints of a prescription drug on a blockchain.

2. A method of claim 1, comprising retrieving active pharmaceuticalingredients (API) and inactive pharmaceutical ingredients (IPI) from theblockchain.

3. A method of claim 2, comprising authenticating the drug aftercomparing the API and IPI with data from Drug Enforcement Administration(DEA) Health and Human Services, (HHS) Food and Drug Administration,(FDA) National Library of Medicines, (NLM) etc. for the purpose ofidentifying the prescription drug'(s) and manufacture name indicated bythose ingredients.4. A method of claim 1, comprising tracing the drug through a supplychain from manufacturer to retailer, dispenser with Pedigree Trail,Serialized Numerical Identification (SNI), Stock Keeping Units (SKU),Point of Sale System (POS) E-Pedigree Systems.5. A method of claim 1, comprising generating a certificate, seal, markand computer scannable symbol such as 2 or 3 dimensional symbol; e.g. QRCode, Bar Code, Watermark, Stealth Dots, etc.Recognition of Exercise Pattern and Tracking of Calorie Consumption

The learning system can be used to detect and monitor user activities asdetected by the ITE sensors. FIG. 8A illustrates the positions of a ski126′ and skier 128′ during a lofting maneuver on the slope 132′. The ski126′ and skier 128′ speed down the slope 132′ and launch into the air136 at position “a,” and later land at position “b” in accord with thewell-known Newtonian laws of physics. With an airtime sensor, describedabove, the unit 10 calculates and stores the total airtime that the ski126′ (and hence the skier 128′) experiences between the positions “a”and “b” so that the skier 128′ can access and assess the “air” timeinformation. Airtime sensors such as the sensor 14 may be constructedwith known components. Preferably, the sensor 14 incorporates either anaccelerometer or a microphone. Alternatively, the sensor 14 may beconstructed as a mechanical switch that detects the presence and absenceof weight onto the switch. Other airtime sensors 14 will become apparentin the description which follows. The accelerometer sensesvibration—particularly the vibration of a vehicle such as a ski ormountain bike—moving along a surface, e.g., a ski slope or mountain biketrail. This voltage output provides an acceleration spectrum over time;and information about airtime can be ascertained by performingcalculations on that spectrum. Based on the information, the system canreconstruct the movement path, the height, the speed, among others andsuch movement data is used to identify the exercise pattern. Forexample, the skier may be interested in practicing mogul runs, and thesystem can identify foot movement and speed and height information andpresent the information post exercises as feedback. Alternatively, thesystem can make live recommendations to improve performance to theathlete.

FIG. 16B illustrates a sensing unit 10″ mounted onto a mountain bike138. FIG. 16B also shows the mountain bike 138 in various positionsduring movement along a mountain bike race course 140 (for illustrativepurposes, the bike 138 is shown without a rider). At one location “c” onthe race course 140, the bike 138 hits a dirt mound 142 and catapultsinto the air 144. The bike 138 thereafter lands at location “d”. Asabove, with speed and airtime sensors, the unit 10 provides informationto a rider of the bike 138 about the speed attained during the ridearound the race course 140; as well as information about the airtimebetween location “c” and “d”. In this case, the system can recommend acadence to be reached by the rider, strengthen of abdominals, back andarms, for example.

For golf exercise, It is beneficial to require the golfer to swing thegolf club a plurality of times at each swing position to account forvariations in each swing. The swing position at which the golf club isswung can be determined by analysis of the measured accelerationprovided by the accelerometer, e.g., the time at which the accelerationchanges. Data obtained during the training stage may be entered into avirtual table of swing positions and estimated carrying distances for aplurality of different swing positions and a plurality of differentswings. A sample format for such a table is as follows, and includes theaveraged carrying distance for each of four different swing positions.The swing analyzer provides a golfer with an excellent estimation of thecarrying distance of a golf ball for a golf club swing at a specificswing position because it has been trained on actual swings by thegolfer of the same club and conversion of information about these swingsinto estimated carrying distances. The golfer can improve their golfgame since they can better select a club to use to hit a golf club fordifferent situations during a round of golf. Also, the swing pattern isused to identify each club path responsible for the curve of any shotand this information is used to improve the golfer. The direction of theclub path relative to the target, out-to-in (fade pattern) or in-to-out(draw pattern), is what I refer to as a players swing pattern. Playersthat swing from in-to-out will tend to hit draws and players that swingfrom out-to-in will tend to hit fades. Where the ball is struck on theface of the driver (strike point) can drastically alter the effect of aplayers swing pattern on ball flight. Thus, the camera detects where theball is struck, and a computer physics model of ball behavior ispresented to the golfer to improve the score. Shots struck off the heelwill tend to fade more or draw less and shots struck off the toe willtend to draw more or fade less. Thus, camera images of the shots struckof heel or toe can also be used to provide patternrecognition/prediction and for training purposes.

For tennis, examples of motions determined for improvement are detailednext. The system can detect if the continental grip is achieved.Throwing Action pattern is also detected, as the tennis serve is anupwards throwing action that would deliver the ball into the air if itwere a baseball pitch. Ball Toss improvements can be determined when theplayer lines the straight arm up with the net post and release the ballwhen your hand reaches eye level. The system checks the forwarddirection so the player can drive weight (and built up momentum) forwardinto the ball and into the direction of the serve.

The sensors can work with a soccer training module with kinematics ofball control, dribbling, passing, crossing, shooting, heading,volleying, taking throw-ins, penalties, corner kicks and free kicks,tackling, marking, juggling, receiving, shielding, clearing, andgoalkeeping. The sensors can work with a basketball training module withkinematics of crossover dribble, behind back, pull back dribble, lowdribble, basic dribble, between legs dribble, Overhead Pass, Chest Pass,Push Pass, Baseball Pass, Off-the-Dribble Pass, Bounce Pass, Jump Shot,Dunk, Free throw, Layup, Three-Point Shot, Hook Shot.

The sensors can work with a baseball training module with kinematics ofHitting, Bunting, Base Running and Stealing, Sliding, Throwing, FieldingGround Balls, Fielding Fly Balls, Double Plays and Relays, Pitching andCatching, Changing Speeds, Holding Runners, Pitching and PitcherFielding Plays, Catching and Catcher Fielding Plays.

For weight training, the sensor can be in gloves as detailed above, orcan be embedded inside the weight itself, or can be in a smart watch,for example. The user would enter an app indicating that the user isdoing weight exercises and the weight is identified as a dumbbell, acurl bar, and a bar bell. Based on the arm or leg motion, the systemautomatically detects the type of weight exercise being done. In oneembodiment shown in FIG. 15C, with motion patterns captured by glove andsock sensors, the system can automatically detect the followingexemplary exercise:

Upper Body:

Chest: Barbell Bench Presses, Barbell Incline Presses, Dumbbell BenchPresses, Dumbbell Incline Presses, Dumbbell Flyes, Cable Crossovers

Back: Pull-Ups, Wide-Grip Lat Pulldowns, One-Arm Dumbbell Rows, SeatedCable Rows, Back Extensions, Straight Arm Pulldowns

Shoulders: Seated Dumbbell Presses, Front Raises, Lateral Raises,Reverse Flyes, Upright Cable Rows, Upright Barbell Rows

Biceps: Alternate Dumbbell Curls, Barbell Curls, Preacher Curls,Concentration Curls, Cable Curls, Hammer Curls

Triceps: Seated Triceps Presses, Lying Triceps Presses, TricepsKickbacks, Triceps Pushdowns, Cable Extensions, Bench Dips

Lower Body

Quadriceps: Barbell Squats, Leg Presses, Leg Extensions

Hamstrings: Dumbbell Lunges, Straight-Leg Deadlifts, Lying Leg Curls

Calves: Seated Calf Raises, Standing Heel Raises

Abs: Floor Crunches, Oblique Floor Crunches, Decline Crunches, DeclineOblique, Hanging Knee Raises, Reverse Crunches, Cable Crunches, CableOblique Crunches

In one implementation in FIG. 16D, an HMM is used to track weightliftingmotor skills or sport enthusiast movement patterns. Human movementinvolves a periodic motion of the legs. Regular walking involves thecoordination of motion at the hip, knee and ankle, which consist ofcomplex joints. The muscular groups attached at various locations alongthe skeletal structure often have multiple functions. The majority ofenergy expended during walking is for vertical motion of the body. Whena body is in contact with the ground, the downward force due to gravityis reflected back to the body as a reaction to the force. When a personstands still, this ground reaction force is equal to the person's weightmultiplied by gravitational acceleration. Forces can act in otherdirections. For example, when we walk, we also produce friction forceson the ground. When the foot hits the ground at a heel strike, thefriction between the heel and the ground causes a friction force in thehorizontal plane to act backwards against the foot. This force thereforecauses a breaking action on the body and slows it down. Not only dopeople accelerate and brake while walking, they also climb and dive.Since reaction force is mass times acceleration, any such accelerationof the body will be reflected in a reaction when at least one foot is onthe ground. An upwards acceleration will be reflected in an increase inthe vertical load recorded, while a downwards acceleration will bereduce the effective body weight. Zigbee wireless sensors with tri-axialaccelerometers are mounted to the sport enthusiast on different bodylocations for recording, for example the tree structure as shown in FIG.16D. As shown therein, sensors can be placed on the four branches of thelinks connect to the root node (torso) with the connected joint, leftshoulder (LS), right shoulder (RS), left hip (LH), and right hip (RH).Furthermore, the left elbow (LE), right elbow (RE), left knee (LK), andright knee (RK) connect the upper and the lower extremities. Thewireless monitoring devices can also be placed on upper back body nearthe neck, mid back near the waist, and at the front of the right legnear the ankle, among others.

The sequence of human motions can be classified into several groups ofsimilar postures and represented by mathematical models calledmodel-states. A model-state contains the extracted features of bodysignatures and other associated characteristics of body signatures.Moreover, a posture graph is used to depict the inter-relationshipsamong all the model-states, defined as PG(ND,LK), where ND is a finiteset of nodes and LK is a set of directional connections between everytwo nodes. The directional connection links are called posture links.Each node represents one model-state, and each link indicates atransition between two model-states. In the posture graph, each node mayhave posture links pointing to itself or the other nodes.

In the pre-processing phase, the system obtains the human body profileand the body signatures to produce feature vectors. In the modelconstruction phase, the system generate a posture graph, examinefeatures from body signatures to construct the model parameters of HMM,and analyze human body contours to generate the model parameters ofASMs. In the motion analysis phase, the system uses features extractedfrom the body signature sequence and then applies the pre-trained HMM tofind the posture transition path, which can be used to recognize themotion type. Then, a motion characteristic curve generation procedurecomputes the motion parameters and produces the motion characteristiccurves. These motion parameters and curves are stored over time, and ifdifferences for the motion parameters and curves over time is detected,the system then runs the sport enthusiast through additional tests toconfirm the detected motion.

In one exemplary process for determining exercise in the left or righthalf of the body, the process compares historical left shoulder (LS)strength against current LS strength (3200). The process also compareshistorical right shoulder (RS) strength against current RS strength(3202). The process can compare historical left hip (LH) strengthagainst current LH strength (3204). The process can also comparehistorical right hip (RH) strength against current RH strength (3206).If the variance between historical and current strength exceedsthreshold, the process generates warnings (3208). Furthermore, similarcomparisons can be made for sensors attached to the left elbow (LE),right elbow (RE), left knee (LK), and right knee (RK) connect the upperand the lower extremities, among others.

The system can ask the sport enthusiast to squeeze a strength gauge,piezoelectric sensor, or force sensor to determine force applied duringsqueeze. The user holds the sensor or otherwise engages the sensor. Theuser then applies and holds a force (e.g., compression, torque, etc.) tothe sensor, which starts a timer clock and triggers a sampling startindicator to notify the user to continue to apply (maximum) force to thesensor. Strength measurements are then sampled periodically during thesampling period until the expiration of time. From the sampled strengthdata, certain strength measurement values are selected, such as themaximum value, average value(s), or values obtained during the samplingperiod. The user can test both hands at the same time, or alternativelyhe may test one hand at a time. A similar approach is used to sense legstrength, except that the user is asked to pushed down on a scale todetermine the foot force generated by the user.

In one embodiment, exercise motion data acquired by the accelerometer ormulti-axis force sensor is analyzed, as will be discussed below, inorder to determine the motion of each exercise stroke during theexercise session (i.e., horizontal vertical or circular). In anotherembodiment for detecting exercise motion using accelerometer, the firstminimum discovered during the scanning is noted as the first xmin andconsidered to be the start of the first brushstroke. The first maximum xvalue following the first minimum x value is located and construed to bethe middle of the first exercise stroke (where exercise motion changesfrom one direction to the other). The next xmin value indicates the endof the first brushstroke and the beginning of the next brushstroke. Thecomputer records the data for each brushstroke and continues on throughthe data to find the next brushstroke, recording each successive motionin memory. For the first brushstroke, the maximum and minimum values ofthe x coordinate (xmax and xmin) are determined. The Y-directionlengths, Ly1 and Ly2, between the data points just before and just aftereach of xmax and xmin (xmax+1, xmax−1, and Xmin+1, xmin−1) are thendetermined. The length Lx along the x axis, between xmax and xmin, isalso determined. Next, if Lx is less than 2 and either Ly1 or Ly2 isgreater than one, then the motion is construed to be vertical. If Ly1and Ly2 are both less than one, then the motion is construed to behorizontal. Otherwise, the motion is construed to be circular.

Data obtained from the gyroscope, if one is used, typically does notrequire a complex analysis. To determine which side of the mouth isbeing brushed at a particular time, the gyroscope data is scanned todetermine when the rotational orientation is greater than 180 degrees,indicating the left side, and when it is less than 180 degrees,indicating the right side. As explained above, top and bottom and gumbrushing information can also be obtained, without any calculations,simply by examining the data. The time sequence of data that is acquiredduring exercise and analyzed as discussed above can be used in a widevariety of ways.

In one embodiment, the accelerometers distinguish between lying down andeach upright position of sitting and standing based on the continuousoutput of the 3D accelerometer. The system can detect (a) extended timein a single position; (b) extended time sitting in a slouching posture(kyphosis) as opposed to sitting in an erect posture (lordosis); and (c)repetitive stressful movements, such as may be found on somemanufacturing lines, while typing for an extended period of time withoutproper wrist support, or while working all day at a weight liftingexercise, among others. In one alternative embodiment, angular positionsensors, one on each side of the hip joint, can be used to distinguishlying down, sitting, and standing positions. In another embodiment, thesystem repeatedly records position and/or posture data over time. In oneembodiment, magnetometers can be attached to a thigh and the torso toprovide absolute rotational position about an axis coincident withEarth's gravity vector (compass heading, or yaw). In another embodiment,the rotational position can be determined through the in-doorpositioning system as discussed above.

To improve a golf swing, the complex motion of the body first startswith the stance. The system checks that the golfer has a low center ofgravity to remain balanced throughout the swing path. The swing startswith the arms moving back in a straight line. When the club head reachesthe level of the hip, two things happen: there is a stern wrist cockthat acts as a hinge along with the left knee (for a right handedswing), building up its torque by moving into the same line as the bellybutton before the start of the upswing. As the swing continues to thetop of the backswing (again for right handed golf swing), the golfer'sleft arm should be perfectly straight and his right arm should be hingedat the elbow. The downswing begins with the hips and the lower bodyrather than the arms and upper body, with emphasis on the wrist cock. Asthe golfer's hips turn into the shot, the right elbow will drop straightdown, hugging the right side of the golfer's torso. As the right elbowdrops, the wrists begin to snap through from the wrist cock in thebackswing. A solid extension of the arms and good transfer of bodyshould put the golfer leaning up on his right toe, balanced, with thegolf club resting on the back of the golfers neck. Importantly, all ofthe movements occur with precise timing, while the head remainscompletely still with eyes focused on the ball throughout the entireswing.

The system can identify illnesses and prevent overexertion leading toillnesses such as a stroke. Depending on the severity of the stroke,sport enthusiasts can experience a loss of consciousness, cognitivedeficits, speech dysfunction, limb weakness, hemiplegia, vertigo,diplopia, lower cranial nerve dysfunction, gaze deviation, ataxia,hemianopia, and aphasia, among others. Four classic syndromes that arecharacteristically caused by lacunar-type stroke are: pure motorhemiparesis, pure sensory syndrome, ataxic hemiparesis syndrome, andclumsy-hand dysarthria syndrome. Sport enthusiasts with pure motorhemiparesis present with face, arm, and leg weakness. This conditionusually affects the extremities equally, but in some cases it affectsone extremity more than the other. The most common stroke location inaffected sport enthusiasts is the posterior limb of the internalcapsule, which carries the descending corticospinal and corticobulbarfibers. Other stroke locations include the pons, midbrain, and medulla.Pure sensory syndrome is characterized by hemibody sensory symptoms thatinvolve the face, arm, leg, and trunk. It is usually the result of aninfarct in the thalamus. Ataxic hemiparesis syndrome features acombination of cerebellar and motor symptoms on the same side of thebody. The leg is typically more affected than the arm. This syndrome canoccur as a result of a stroke in the pons, the internal capsule, or themidbrain, or in the anterior cerebral artery distribution. Sportenthusiasts with clumsy-hand dysarthria syndrome experience unilateralhand weakness and dysarthria. The dysarthria is often severe, whereasthe hand involvement is more subtle, and sport enthusiasts may describetheir hand movements as “awkward.” This syndrome is usually caused by aninfarct in the pons. Different patterns of signs can provide clues as toboth the location and the mechanism of a particular stroke. The systemcan detect symptoms suggestive of a brainstem stroke include vertigo,diplopia, bilateral abnormalities, lower cranial nerve dysfunction, gazedeviation (toward the side of weakness), and ataxia. Indications ofhigher cortical dysfunction-such as neglect, hemianopsia, aphasia, andgaze preference (opposite the side of weakness)—suggest hemisphericdysfunction with involvement of a superficial territory from anatherothrombotic or embolic occlusion of a mainstem vessel or peripheralbranch.

To detect muscle weakness or numbness, in one embodiment, the systemapplies a pattern recognizer such as a neural network or a Hidden MarkovModel (HMM) to analyze accelerometer output. In another embodiment,electromyography (EMG) is used to detect muscle weakness. In anotherembodiment, EMG and a pattern analyzer is used to detect muscleweakness. In yet another embodiment, a pattern analyzer analyzes bothaccelerometer and EMG data to determine muscle weakness. In a furtherembodiment, historical ambulatory information (time and place) is usedto further detect changes in muscle strength. In yet other embodiments,accelerometer data is used to confirm that the sport enthusiast is atrest so that EMG data can be accurately captured or to compensate formotion artifacts in the EMG data in accordance with a linear ornon-linear compensation table. In yet another embodiment, the EMG datais used to detect muscle fatigue and to generate a warning to the sportenthusiast to get to a resting place or a notification to a nurse orcaregiver to render timely assistance. The amplitude of the EMG signalis stochastic (random) in nature and can be reasonably represented by aGausian distribution function. The amplitude of the signal can rangefrom 0 to 10 mV (peak-to-peak) or 0 to 1.5 mV (rms). The usable energyof the signal is limited to the 0 to 500 Hz frequency range, with thedominant energy being in the 50-150 Hz range. Usable signals are thosewith energy above the electrical noise level. The dominant concern forthe ambient noise arises from the 60 Hz (or 50 Hz) radiation from powersources. The ambient noise signal may have an amplitude that is one tothree orders of magnitude greater than the EMG signal. There are twomain sources of motion artifact: one from the interface between thedetection surface of the electrode and the skin, the other from movementof the cable connecting the electrode to the amplifier. The electricalsignals of both noise sources have most of their energy in the frequencyrange from 0 to 20 Hz and can be reduced.

In one embodiment, the camera captures facial expression and a code suchas the Microsoft Emotion API takes a facial expression in an image as aninput, and returns the confidence across a set of emotions for each facein the image, as well as bounding box for the face, using the Face API.The emotions detected are anger, contempt, disgust, fear, happiness,neutral, sadness, and surprise. These emotions are understood to becross-culturally and universally communicated with particular facialexpressions. Alternatively, a marker for emotional arousal is galvanicskin response (GSR), also referred to as skin conductance (SC) orelectro-dermal activity (EDA). EDA modulates the amount of sweatsecretion from sweat glands. The amount of sweat glands varies acrossthe human body, being highest in hand and foot regions (200-600 sweatglands per cm2). While sweat secretion plays a major role forthermoregulation and sensory discrimination, changes in skin conductancein hand and foot regions are also triggered quite impressively byemotional stimulation: the higher the arousal, the higher the skinconductance. It is noteworthy to mention that both positive (“happy” or“joyful”) and negative (“threatening” or “saddening”) stimuli can resultin an increase in arousal—and in an increase in skin conductance. Skinconductance is not under conscious control. Instead, it is modulatedautonomously by sympathetic activity which drives human behavior,cognitive and emotional states on a subconscious level. Skin conductancetherefore offers direct insights into autonomous emotional regulation.It can be used as alternative to self-reflective test procedures,or—even better—as additional source of insight to validate verbalself-reports or interviews of a respondent. Based on the detectedemotion, the exercise can be increased, decreased, or stoppedaltogether.

Data from multiple exercise sessions may be collected and used tocompile a history of the user's habits over an extended period of time,enabling the user's trainer to better understand user compliance issues.The trainer can review the data with the user and view the animations ofthe user's exercise sessions during an office visit, allowing thetrainer to better instruct the user in proper brushing technique. Thetrainer can also review the patient's brushing history over time, todetermine whether the patient's exercise technique is improving.

The sensor can be integrated into objects already associated with thesporting activity. In one aspect, the sensing unit is integrated intothe ski boot or other boot. In another aspect, the sensing unit isintegrated into the binding for a ski boot or snowboarder boot. In stillanother aspect, the sensing unit is integrated into a ski, snowboard,mountain bike, windsurfer, windsurfer mast, roller blade boot,skate-board, kayak, or other sport vehicle. Collectively, the sportobjects such as the ski boot and the variety of sport vehicles aredenoted as “sport implements”. Accordingly, when the sensing unit is not“stand alone”, the housing which integrates the controller subsystemwith one or more sensors and battery can be made from the material ofthe associated sport implement, in whole or in part, such that thesensing unit becomes integral with the sport implement.

A mobile app may be a computer implemented method. A computer programmay be provided for executing the app with code for:

(21) capture user motion with accelerometer or gyroscope

(22) capture VR views through camera and process using GPU

(23) capture user emotion using facial recognition or GSR

(24) model user action using kinematic model

(25) compare user action with idea action

(26) coach user on improvement to user sport techniques.

The device can negotiate and enforce agreements with others blockchainsmart contracts. The system may include one or more of the following:

code to determine trade settlement amounts and transfers fundsautomatically,

code to automatically pay coupon payments and returns principal uponbond expiration,

code to determine payout based on claim type and policy coverage,

code to collect insurance based on usage and upon a claim submission,code to determine payout based on claim type and policy coverage,

code to transfer electronic medical record from a source to adestination based on patient consent,

code to anonymously store wearable health data from wearable devices forpublic health monitoring,

a secured content and code to determine and distributes royalty to anauthor,

code for storing a stock certificate number with stock quantity,

code to determine a share registry or a capitalization table from eachstock certificate number and stock quantity,

code to distribute shareholder communication from a share registry or acapitalization table,

code to collect secure shareholder votes from a share registry or acapitalization table for transparent corporate governance,

code to provide financial information to shareholder a share registry ora capitalization table for corporate governance,

code to enforce majority or supermajority shareholder votes from a shareregistry or a capitalization table for corporate governance,

code for supply chain management,

code for tracking chain of custody for an item, or

code for peer-to-peer transactions for between two computers.

Although described with respect to ear bud and flexible ear enclosures,the sensor emitter and detector may be enclosed in any number ofhousings having various sizes and shapes of ear tissue contact surfaces,may use various types of electrical interconnnect and use variousmaterials so as to noninvasively measure blood parameters from the ear.

The system may include modifying a characteristic of thethree-dimensional model to improve the fit. This may include modifying ashell for an earpiece, modifying a shape of the earpiece, selectingdifferent (e.g., firmer or softer) materials for fabrication of theearpiece or otherwise modifying a material profile of the 3D model, andso forth. Modifying the characteristic may also or instead includepositioning an articulating joint within the three-dimensional model,e.g., to accommodate axial deformation during insertion/removal of theearpiece. Modifying the characteristic may also or instead includemodifying an elasticity of a portion of the three-dimensional model.

It will further be appreciated that, based on the compliance datacaptured during a scan, a good estimate can be obtained of the maximumshort-duration expansion of regions of the ear canal. This data may beuseful for modeling the insertion and removal of the earpiece, andmodifying the earpiece design accordingly to reduce discomfort duringinsertion and removal of the earpiece.

It will be readily appreciated that a device such as any of the devicesdescribed above may be adapted to perform the method with suitableprogramming or other configuration of the processor and/or otherprocessing circuitry. Also disclosed herein is a computer programproduct comprising computer executable code embodied in a non-transitorycomputer readable medium that, when executing on one or more computingdevices, performs the processing steps associated with the method.

It will be appreciated that any of the above systems, devices, methods,processes, and the like may be realized in hardware, software, or anycombination of these suitable for the control, data acquisition, anddata processing described herein. This includes realization in one ormore microprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable devices,along with internal and/or external memory. This may also, or instead,include one or more application specific integrated circuits,programmable gate arrays, programmable array logic components, or anyother device or devices that may be configured to process electronicsignals. It will further be appreciated that a realization of theprocesses or devices described above may include computer-executablecode created using a structured programming language such as C, anobject oriented programming language such as C++, or any otherhigh-level or low-level programming language (including assemblylanguages, hardware description languages, and database programminglanguages and technologies) that may be stored, compiled or interpretedto run on one of the above devices, as well as heterogeneouscombinations of processors, processor architectures, or combinations ofdifferent hardware and software. At the same time, processing may bedistributed across devices such as a camera and/or computer and/orserver or other remote processing resource in a number of ways, or allof the functionality may be integrated into a dedicated, standalonedevice. All such permutations and combinations are intended to fallwithin the scope of the present disclosure.

In other embodiments, disclosed herein are computer program productscomprising computer-executable code or computer-usable code that, whenexecuting on one or more computing devices, performs any and/or all ofthe steps described above. The code may be stored in a computer memory,which may be a memory from which the program executes (such as randomaccess memory associated with a processor), or a storage device such asa disk drive, flash memory or any other optical, electromagnetic,magnetic, infrared or other device or combination of devices. In anotheraspect, any of the processes described above may be embodied in anysuitable transmission or propagation medium carrying thecomputer-executable code described above and/or any inputs or outputsfrom same.

While the invention has been disclosed in connection with the preferredembodiments shown and described in detail, various modifications andimprovements thereon will become readily apparent to those skilled inthe art. Accordingly, the spirit and scope of the preferred embodimentis not to be limited by the foregoing examples, but is to be understoodin the broadest sense allowable by law.

What is claimed is:
 1. A method for assisting a user, comprising:customizing an in-ear device to a user anatomy; capturing sound usingthe in-ear device; amplifying sound using an amplifier with gain andamplitude controls for a plurality of frequencies; and applying alearning machine to identify an aural environment and adjust theamplifiers for optimum hearing.
 2. The method of claim 1, comprisingapplying a learning machine to adjust amplifier parameters for aparticular environment with a predetermined noise pattern.
 3. The methodof claim 1, comprising capturing vital signs with the in-ear device witha camera in the ear to detect ear health.
 4. The method of claim 1,comprising capturing vital signs with the in-ear device by detectingblood flow with an in-ear sensor.
 5. The method of claim 1, comprisingcapturing vital signs with the in-ear device by detecting with an in-earsensor blood parameters including carboxyhemoglobin (HbCO),methemoglobin (HbMet) and total hemoglobin (Hbt).
 6. The method of claim1, comprising detecting a physical condition of an ear drum includingcurvature and surface abnormality.
 7. The method of claim 1, comprisingcapturing vital signs with the in-ear device by detecting bodytemperature, encephalography (EEG) data, electrocardiogram (ECG) data,or bioimpedance (BI) data in the ear.
 8. The method of claim 1,comprising capturing vital signs with the in-ear device by detecting oneor more of: alpha rhythm, auditory steady-state response (ASSR),steady-state visual evoked potentials (SSVEP), visually evoked potential(VEP), visually evoked response (VER) and visually evoked corticalpotential (VECP), cardiac activity, bioimpedance, speech and breathing.9. The method of claim 1, comprising detecting alpha rhythm, auditorysteady-state response (ASSR), steady-state visual evoked potentials(SSVEP), and visually evoked potential (VEP).
 10. The method of claim 1,comprising correlating EEG, ECG, bioimpedance, speech and breathing todetermine health.
 11. The method of claim 1, comprising correlatingcardiac activity, bioimpedance and breathing.
 12. The method of claim 1,comprising determining user health by detecting fluid in an earstructure, change in ear color, curvature of the ear structure.
 13. Themethod of claim 1, comprising determining one or more bio-markers fromthe vital signs and indicating user health.
 14. The method of claim 1,wherein the customizing comprises performing a 3D scan inside an earcanal.
 15. The method of claim 14, comprising matching predeterminedpoints on the 3D scan to key points on a template and morphing the keypoints on the template to the predetermined points.
 16. The method ofclaim 1, comprising 3D printing a model from the 3D scan and fabricatingthe in-ear device.
 17. The method of claim 1, comprising correlatinggenomic biomarkers for diseases to the vital signs from the in-eardevice and applying a learning machine to use the vital signs from thein-ear device to predict disease conditions.
 18. The method of claim 1,comprising determining a fall based on accelerometer data, vital signsand sound.
 19. The method of claim 1, comprising determining lonelinessor mental condition based on activity of life data.
 20. The method ofclaim 1, comprising providing a user dashboard showing health data overa period of time and matching research data on the health signals. 21.The method of claim 1, comprising: providing a system comprising: ahousing custom fitted to a user anatomy; a microphone to capture soundcoupled to a processor to deliver enhanced sound to the user anatomy; anamplifier with gain and amplitude controls; and a learning machine toidentify an aural environment and adjusting amplifier controls tooptimize hearing based on the identified aural environment.